Fully Automatic Fingerprint Minutiae Extraction Method and System

Abstract

The present invention provides a fully automatic fingerprint minutiae feature extraction method and system, including: Step S1: forming a fingerprint minutiae extraction network D based on fingerprint prior knowledge and a fingerprint minutiae prediction network; Step S2: preprocessing a fingerprint image to obtain a preprocessed fingerprint image; Step S3: training the fingerprint minutiae extraction network D with the preprocessed fingerprint image to obtain a trained fingerprint minutiae extraction network D; Step S4: preliminarily predicting a fingerprint minutiae point set with the trained fingerprint minutiae extraction network D; Step S5: eliminating fingerprint redundant points from the preliminarily predicted fingerprint minutiae point set by applying non-maximum suppression based on the general intersection over union to obtain a final accurate fingerprint minutiae point set.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing and pattern recognition, and in particular to a method for fully automatic fingerprint detail feature point extraction. Background Art

[0002] In recent years, with the great progress of network computing and services, GPU resource virtualization, and system integration technology, it has become possible for online automatic fingerprint recognition technology to be widely used in the real world. Fingerprint recognition technology plays an important role in criminal investigation, forensic identification, finance, and social security. Generally speaking, fingerprint recognition technology includes two stages: registration and identification and authentication. The fingerprint registration stage mainly involves fingerprint image acquisition, foreground image segmentation, normalization, directional field / frequency field estimation, enhancement, binarization, and refinement; while the identification and authentication stage usually includes two processes: feature point extraction and matching, that is, finding the correspondence between the detail points in the registered fingerprint and the detail points in the query fingerprint. Among them, fingerprint detail point extraction is the basic work of fingerprint recognition. Accurate fingerprint detail point extraction is the premise of subsequent matching and authentication. This step can be abstracted as a classic pattern recognition problem. However, when collecting fingerprints, the fingerprint image is usually degraded due to the noise of the collection equipment and the uneven force of the collector. These factors increase the complexity of recognition. Therefore, effective modeling of degraded fingerprint images is a problem that has not been effectively solved so far.

[0003] In the past decade, deep neural networks have been widely used in pattern recognition and computer vision tasks, especially in target detection, showing impressive results. Fingerprint detail detection can be summarized as a small target detection task. In addition, with the maturity of GPU resource virtualization technology, the training and reasoning of deep learning models can be promoted under non-laboratory conditions. In real-world applications, the fingerprint detail feature reasoning process does not require much GPU computing power compared to training. Therefore, how to improve the accuracy and completeness of the model for detail point detection is of great significance to actual application scenarios and is also one of the problems that need to be solved urgently.

[0004] Patent document CN103824060A (application number: 201410073986.5) discloses a method for extracting minutiae points of a fingerprint, comprising the following steps: performing multi-channel image enhancement on a received fingerprint to obtain N enhanced images; calibrating the direction field of the fingerprint, and dividing the fingerprint into N fingerprint regions according to the direction field; extracting minutiae points of the enhanced images from the N enhanced images respectively; obtaining minutiae points corresponding to the N fingerprint regions from the N enhanced images respectively, and combining the minutiae points in the N fingerprint regions to obtain a set of candidate minutiae points of the fingerprint; and removing overlapping minutiae points in the set of candidate minutiae points to obtain a final set of minutiae points. Summary of the invention

[0005] Aiming at the defects in the prior art, the purpose of the present invention is to provide a fully automatic fingerprint minutiae extraction method and system.

[0006] A fully automatic fingerprint minutiae extraction method provided by the present invention includes:

[0007] Step S1: Based on fingerprint prior knowledge and a fingerprint minutiae preliminary prediction network, a fingerprint minutiae extraction network D is formed;

[0008] Step S2: Preprocess the fingerprint image to obtain the preprocessed fingerprint image;

[0009] Step S3: Use the preprocessed fingerprint image to train the fingerprint minutiae extraction network D to obtain the trained fingerprint minutiae extraction network D;

[0010] Step S4: Use the trained fingerprint minutiae extraction network D to preliminarily predict the fingerprint minutiae point set;

[0011] Step S5: Apply the non-maximum suppression algorithm based on the general intersection over union to the preliminarily predicted fingerprint minutiae point set to remove fingerprint redundant points, and obtain the final accurate fingerprint minutiae point set;

[0012] The fingerprint minutiae preliminary prediction network is a convolutional neural network for direction prediction, fingerprint segmentation, and minutiae extraction based on ResNet;

[0013] The fingerprint minutiae extraction network D is a minutiae detection method based on the FingerNet framework; it includes using a residual network to predict the fingerprint direction field for fingerprint image segmentation and using a residual network to predict fingerprint minutiae points, and finally deleting redundant points based on the proposed general intersection over union non-maximum suppression algorithm.

[0014] Preferably, the step S2 adopts:

[0015] Adopt a per-pixel normalization method for the fingerprint image to adjust each pixel to a fixed ratio and interval;

[0016]

[0017] Among them, I(i,j) represents the image intensity of the input image I at the pixel (i,j); Mean and Var respectively represent the image mean and variance; M0 and Var0 respectively represent the image mean and variance after normalization.

[0018] Preferably, the step S3 adopts: The preprocessed fingerprint image passes through the fingerprint minutiae extraction network D to generate a predicted minutiae point sequence P=(M p ), and is the same as the original labeled ground truth minutiae points G=(M g)Calculate the difference loss function L of the attribute information of the fingerprint minutiae and backpropagate for optimization iteration to obtain the trained fingerprint minutiae extraction network D; where M p represents the predicted minutiae set, and M g represents the ground-truth minutiae set corresponding to the labeled fingerprint image.

[0019] Preferably, the fingerprint minutiae extraction network D includes: a fingerprint orientation estimation and fingerprint image segmentation module composed of ResNet, a phase filter set module, an enhanced fingerprint module, and a ResNet minutiae extraction module;

[0020] Step S3.1: Pass the preprocessed fingerprint image through the phase filter set module composed of 2D convolutions to obtain a fingerprint phase feature map;

[0021] Step S3.2: Pass the original fingerprint image through the phase filter set module composed of 2D convolutions to obtain an amplitude feature map;

[0022] Step S3.3: Pass the preprocessed fingerprint image through the fingerprint orientation estimation and fingerprint image segmentation module composed of ResNet to obtain a fingerprint orientation feature map and a fingerprint segmentation feature map;

[0023] Step S3.4: Pass the obtained fingerprint phase feature map, fingerprint orientation feature map, and amplitude feature map through the enhanced fingerprint module for fingerprint enhancement to obtain an enhanced feature map;

[0024] Step S3.5: Perform concatenation and merging operations on the fingerprint segmentation feature map and the enhanced feature map to obtain a fused fingerprint feature map;

[0025] Step S3.6: Pass the fused fingerprint feature map through the ResNet minutiae extraction module to predict the final feature point information and generate predicted minutiae;

[0026] Step S3.7: Calculate the objective loss function L between the predicted minutiae and the original labeled ground-truth minutiae, and use the backpropagation algorithm to optimize the objective loss L. Repeat steps S3.1 to S3.7 until the objective function converges to obtain the trained fingerprint minutiae extraction network D.

[0027] Preferably, step S5 adopts:

[0028] The calculation formula of the general intersection over union is as follows:

[0029] Suppose there are two arbitrary shapes E and F. Find a smallest closed shape G that contains E and F. Then calculate the proportion of the area in G that does not cover E and F to the area of G. Finally, subtract the proportion of the area in G that does not cover E and F to the area of G from the IoU of E and F:

[0030]

[0031] Among them, As an evaluation index for whether it is an abnormal point.

[0032] A fully automatic fingerprint minutiae extraction system provided by the present invention includes:

[0033] Module M1: Based on fingerprint prior knowledge and a fingerprint minutiae preliminary prediction network to form a fingerprint minutiae extraction network D;

[0034] Module M2: Preprocess the fingerprint image to obtain a preprocessed fingerprint image;

[0035] Module M3: Use the preprocessed fingerprint image to train the fingerprint minutiae extraction network D to obtain a trained fingerprint minutiae extraction network D;

[0036] Module M4: Use the trained fingerprint minutiae extraction network D to preliminarily predict a fingerprint minutiae point set;

[0037] Module M5: Apply the non-maximum suppression algorithm based on the general intersection over union to the preliminarily predicted fingerprint minutiae point set to remove fingerprint redundant points, and obtain a final accurate fingerprint minutiae point set;

[0038] The fingerprint minutiae preliminary prediction network is a convolutional neural network based on ResNet for direction prediction, fingerprint segmentation, and minutiae extraction;

[0039] The fingerprint minutiae extraction network D is a minutiae detection method based on the FingerNet framework; it includes using a residual network to predict the fingerprint direction field for fingerprint image segmentation and using a residual network to predict fingerprint minutiae points, and finally deleting redundant points based on the proposed general intersection over union non-maximum suppression algorithm.

[0040] Preferably, the module M2 adopts:

[0041] Adopt a per-pixel normalization method for the fingerprint image to adjust each pixel to a fixed ratio and interval;

[0042]

[0043] Among them, I(i,j) represents the image intensity of the input image I at the pixel (i,j); Mean and Var respectively represent the image mean and variance; M0 and Var0 respectively represent the image mean and variance after normalization.

[0044] Preferably, the module M3 adopts: The preprocessed fingerprint image passes through the fingerprint minutiae extraction network D to generate a predicted minutiae point sequence P=(M p ), and is the same as the original labeled ground truth minutiae G=(Mg ) Calculate the difference loss function L of attribute information of fingerprints and backpropagate for optimization iteration to obtain the trained fingerprint minutiae extraction network D; where M p represents the predicted minutiae set, and M g represents the ground-truth minutiae set corresponding to the labeled fingerprint image.

[0045] Preferably, the fingerprint minutiae extraction network D includes: a fingerprint orientation estimation and fingerprint image segmentation module, a phase filter set module, an enhanced fingerprint module, and a ResNet minutiae extraction module composed of ResNet;

[0046] Module M3.1: Obtain the fingerprint phase feature map by passing the preprocessed fingerprint image through the phase filter set module composed of 2D convolutions;

[0047] Module M3.2: Obtain the amplitude feature map by passing the original fingerprint image through the phase filter set module composed of 2D convolutions;

[0048] Module M3.3: Pass the preprocessed fingerprint image through the fingerprint orientation estimation and fingerprint image segmentation module composed of ResNet to obtain the fingerprint orientation feature map and the fingerprint segmentation feature map;

[0049] Module M3.4: Enhance the fingerprint by passing the obtained fingerprint phase feature map, fingerprint orientation feature map, and amplitude feature map through the enhanced fingerprint module to obtain the enhanced feature map;

[0050] Module M3.5: Perform concatenation and merging operations on the fingerprint segmentation feature map and the enhanced feature map to obtain the fused fingerprint feature map;

[0051] Module M3.6: Predict the final feature point information through the ResNet minutiae extraction module for the fused fingerprint feature map to generate the predicted minutiae;

[0052] Module M3.7: Calculate the objective loss function L between the predicted minutiae and the original labeled ground-truth minutiae, and use the backpropagation algorithm to optimize the objective loss L. Repeat triggering Module M3.1 to Module M3.7 until the objective function converges to obtain the trained fingerprint minutiae extraction network D.

[0053] Preferably, the module M5 adopts:

[0054] The calculation formula of the general intersection over union is as follows:

[0055] Suppose there are two arbitrary shapes E and F. Find a smallest closed shape G such that G contains E and F. Then calculate the proportion of the area in G that does not cover E and F to the area of G. Finally, subtract the proportion of the area in G that does not cover E and F to the area of G from the IoU of E and F:

[0056]

[0057] Among them, As an evaluation index for whether it is an abnormal point.

[0058] Compared with the prior art, the present invention has the following beneficial effects:

[0059] 1. The present invention proposes a fingerprint minutiae detection method based on the ResNet structure, which can obtain a more efficient fingerprint minutiae detection method compared with the past general CNN structure;

[0060] 2. The present invention uses an efficient neural network architecture to simulate fingerprint orientation field estimation and fingerprint segmentation operations, and can obtain richer fingerprint features. In order to achieve high-precision fingerprint minutiae extraction, the present invention also introduces a residual structure (direct connection path) into the high-speed path network, and deepens the depth of the fingerprint minutiae extraction network by stacking. The residual structure effectively alleviates the problem of gradient disappearance or explosion during the training process, reduces the optimization complexity, and improves the stability of the model at the same time;

[0061] 3. The present invention first proposes to use a post-processing operation for non-maximum suppression abnormal point deletion based on Generalized Intersection over Union (GIoU), which can efficiently delete redundant points in the initial detected point set;

[0062] 4. The software developed by using the method of the present invention has the advantages of good detection effect, being convenient for deployment in real-world applications, simple operation, etc., and is more suitable for the actual situation of fingerprint recognition and authentication. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] By reading the detailed description of the non-limiting embodiments with reference to the following drawings, other features, objects and advantages of the present invention will become more apparent:

[0064] Figure 1 It is a schematic flowchart of the training and testing of the fingerprint minutiae data extraction method based on deep learning according to the embodiment of the present invention.

[0065] Figure 2 It is a schematic design diagram of the orientation field and segmentation feature extraction network according to the embodiment of the present invention.

[0066] Figure 3 It is a schematic design diagram of the minutiae feature extraction network according to the embodiment of the present invention.

[0067] Figure 4 It is an explanatory diagram of the non-maximum suppression calculation based on Generalized Intersection over Union according to the embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0068] The present invention will be described in detail below with reference to specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but do not limit the present invention in any form. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present invention, several changes and improvements can still be made. These all belong to the protection scope of the present invention.

[0069] Aiming at the problems of low accuracy and integrity in detecting fingerprint minutiae features in the prior art, the purpose of the present invention is to provide a fully automatic fingerprint minutiae feature detection method based on deep learning. In the field of deep learning network architectures, the Residual Network (ResNet) can further increase the network depth compared to general neural network structures in the past, such as LeNet, VGG, etc. By introducing residual layers, the network becomes easier to optimize. Therefore, through comprehensive consideration of both effectiveness and generality, the Residual Network has better performance than general CNNs.

[0070] Example 1

[0071] A fully automatic fingerprint minutiae feature extraction method provided by the present invention includes:

[0072] Step S1: Based on fingerprint prior knowledge and a fingerprint minutiae preliminary prediction network, form a fingerprint minutiae extraction network D;

[0073] Step S2: Preprocess the fingerprint image to obtain the preprocessed fingerprint image;

[0074] Step S3: Use the preprocessed fingerprint image to train the fingerprint minutiae extraction network D to obtain the trained fingerprint minutiae extraction network D;

[0075] Among them, for each unit F=(I) in the training image set, a predicted minutiae point sequence P=(M p ) is generated through the fingerprint minutiae extraction network D, and the attribute information difference loss function L is calculated with the original labeled ground truth minutiae points G=(M g ), and backpropagation is performed for optimization iteration to obtain the optimal fingerprint minutiae extraction network model; where I represents the original fingerprint image in the training set, M p represents the set of predicted minutiae points, and M g represents the set of ground truth minutiae points labeled for the corresponding fingerprint image;

[0076] Step S4: Use the trained fingerprint minutiae extraction network D to preliminarily predict the fingerprint minutiae point set; among them, on the prediction data set, the optimal fingerprint minutiae extraction network model is inferred and applied to preliminarily predict the fingerprint minutiae point set, and the obtained point set is denoted as S naive ;

[0077] Step S5: Apply the non-maximum suppression algorithm based on the general intersection over union (IoU) to the preliminary predicted fingerprint minutiae set to remove redundant fingerprint points, and obtain the final accurate fingerprint minutiae set S final ;

[0078] The preliminary fingerprint minutiae prediction network is a convolutional neural network for direction prediction, fingerprint segmentation, and minutiae extraction based on ResNet;

[0079] The fingerprint minutiae extraction network D is a minutiae detection method based on the FingerNet framework; it includes a residual network for predicting the fingerprint direction field for fingerprint image segmentation, and the current residual unit is denoted as S1; and a residual network for predicting fingerprint minutiae, and the current residual unit is denoted as S2; finally, redundant points are removed based on the proposed general intersection over union non-maximum suppression algorithm.

[0080] The residual unit S1 is used to receive and perform a convolution operation on the input normalized image features, and then extract and output the fingerprint direction distribution features and the fingerprint segmentation mask features through two branches respectively;

[0081] The residual unit S2 is used to receive and take the enhanced image and the segmented image features as inputs, and take the minutiae prediction information as the output, and use the X probability map, the Y probability map, the direction probability map, and the detailed feature confidence score map as the outputs of this module respectively.

[0082] Specifically, the step S2 adopts:

[0083] Adopt a per-pixel normalization method for the fingerprint image to adjust each pixel to a fixed ratio and interval;

[0084]

[0085] Among them, I(i,j) represents the image intensity of the input image I at the pixel (i,j); Mean and Var represent the image mean and variance respectively; M0 and Var0 represent the normalized image mean and variance respectively. The normalization operation here is equivalent to the non-linear activation layer in a general network. The input and output of this layer are a 512×512×1 fingerprint image and a 512×512×1 normalized feature map respectively.

[0086] Specifically, the fingerprint minutiae extraction network D includes: a fingerprint direction estimation and fingerprint image segmentation module, a phase filter set module, an enhanced fingerprint module, and a ResNet minutiae extraction module composed of ResNet;

[0087] The normalized fingerprint feature map is processed in two branches. One branch is fed into the fingerprint orientation estimation and fingerprint image segmentation module composed of ResNet, and the other branch is fed into the phase filter set module composed of 2D convolution. Subsequently, a suitable orientation filter is selected for fingerprint image enhancement.

[0088] Among them, the inputs and outputs of the orientation estimation unit in the ResNet orientation estimation and fingerprint segmentation module are the normalized fingerprint map of 512×512×1 and the orientation feature map of 64×64×90 respectively. The inputs and outputs of the fingerprint segmentation unit in the ResNet orientation estimation and fingerprint segmentation module are the normalized fingerprint image of 512×512×1 and the fingerprint segmentation feature map of 64×64×1 respectively. The input and output of the phase filter set module are the normalized fingerprint image of 512×512×1 and the fingerprint phase feature map of 512×512×90 respectively; Subsequently, the input of the enhanced fingerprint module's complex image amplitude and direction field fusion feature map and the output sizes of the enhanced fingerprint are 512×512×90, 512×512×90, and 512×512×1 respectively.

[0089] ③ Perform concate and merge operations on the fingerprint segmentation feature map and the enhanced feature map. The input and output feature map sizes of this process are (512×512×1, 512×512×1) and 512×512×2 respectively.

[0090] ④ Pass the fused fingerprint feature map through the minutiae feature extraction module composed of ResNet to perform the final minutiae information prediction. The input of this step is the fingerprint structure feature map of 512×512×2, and the outputs are the minutiae direction feature map of 64×64×180, the fingerprint H and W probability coordinate map of size 64×64×8, and the minutiae confidence score map of size 64×64×1.

[0091] ⑤ Calculate the target loss function L (using the calculation mode of cross-entropy and softmax regression) with the original labeled ground truth label and use the backpropagation algorithm for optimization iteration to obtain the optimal model.

[0092] Specifically, step S5 adopts: applying non-maximum suppression based on Generalized Interaction of Union (GIoU) to perform subsequent fingerprint redundant point deletion;

[0093] The calculation formula of the Generalized Interaction of Union is as follows:

[0094] Assume there are two arbitrary shapes E and F, find a smallest closed shape G that contains E and F, then calculate the proportion of the area in G that does not cover E and F to the area of G, and finally subtract the proportion of the area in G that does not cover E and F to the area of G from the IoU of E and F:

[0095]

[0096] Among them, As an evaluation index for whether it is an abnormal point.

[0097] In the subsequent processing steps of the minutiae points, the present invention for the first time uses an evaluation index based on GIoU to perform redundant deletion on the minutiae points inferred by the deep learning model, and obtains a more complete fingerprint minutiae point output.

[0098] According to a fully automatic fingerprint minutiae feature extraction system provided by the present invention, it includes:

[0099] Module M1: Form a fingerprint minutiae extraction network D based on fingerprint prior knowledge and a fingerprint minutiae preliminary prediction network;

[0100] Module M2: Preprocess the fingerprint image to obtain a preprocessed fingerprint image;

[0101] Module M3: Use the preprocessed fingerprint image to train the fingerprint minutiae extraction network D to obtain a trained fingerprint minutiae extraction network D;

[0102] Among them, for each unit F=(I) in the training image set, a predicted minutiae point sequence P=(M p ) is generated through the fingerprint minutiae extraction network D, and the attribute information difference loss function L is calculated with the original labeled ground truth minutiae points G=(M g ) and backpropagated for optimization iteration to obtain an optimal fingerprint minutiae extraction network model; where I represents the original fingerprint image in the training set, M p represents the set of predicted minutiae points, and M g represents the set of ground truth minutiae points labeled for the corresponding fingerprint image;

[0103] Module M4: Use the trained fingerprint minutiae extraction network D to preliminarily predict the fingerprint minutiae point set; among them, on the prediction data set, the optimal fingerprint minutiae extraction network model is inferred and applied to preliminarily predict the fingerprint minutiae point set, and the obtained point set is denoted as S naive ;

[0104] Module M5: Apply the non-maximum suppression algorithm based on the general intersection over union to the preliminarily predicted fingerprint minutiae point set to eliminate fingerprint redundant points, and obtain the final accurate fingerprint minutiae point set S final ;

[0105] The fingerprint minutiae preliminary prediction network is a convolutional neural network based on ResNet for direction prediction, fingerprint segmentation, and minutiae point extraction;

[0106] The fingerprint minutiae extraction network D is a minutiae detection method based on the FingerNet framework; it includes a residual network for predicting the fingerprint orientation field for fingerprint image segmentation, and the current residual unit is denoted as S1; and a residual network for predicting fingerprint minutiae, and the current residual unit is denoted as S2; finally, redundant points are removed based on the proposed general intersection over union non-maximum suppression algorithm.

[0107] The residual unit S1 is used to receive and perform a convolution operation on the input normalized image features, and then extract and output the fingerprint orientation distribution features and the fingerprint segmentation mask features through two branches respectively;

[0108] The residual unit S2 is used to receive and take the enhanced image and the segmentation image features as inputs, and take the minutiae prediction information as the output, and use the X probability map, the Y probability map, the orientation probability map, and the minutiae feature confidence score map as the outputs of this module respectively.

[0109] Specifically, the module M2 adopts:

[0110] Adopt a per-pixel normalization method for the fingerprint image to adjust each pixel to a fixed ratio and interval;

[0111]

[0112] Among them, I(i,j) represents the image intensity of the input image I at the pixel (i,j); Mean and Var represent the image mean and variance respectively; M0 and Var0 represent the normalized image mean and variance respectively. The normalization operation here is equivalent to the non-linear activation layer in a general network. The input and output of this layer are a fingerprint image of 512×512×1 and a normalized feature map of 512×512×1 respectively.

[0113] Specifically, the fingerprint minutiae extraction network D includes: a fingerprint orientation estimation and fingerprint image segmentation module composed of ResNet, a phase filter set module, an enhanced fingerprint module, and a ResNet minutiae extraction module;

[0114] The normalized fingerprint feature map is processed in two branches. One branch is sent to the fingerprint orientation estimation and fingerprint image segmentation module composed of ResNet, and the other branch is sent to the phase filter set module composed of 2D convolution, and then a suitable orientation filter is selected for fingerprint image enhancement.

[0115] Among them, the inputs and outputs of the direction estimation unit in the ResNet direction estimation and fingerprint segmentation module are a normalized fingerprint map of 512×512×1 and a direction feature map of 64×64×90 respectively. The inputs and outputs of the fingerprint segmentation unit in the ResNet direction estimation and fingerprint segmentation module are a normalized fingerprint image of 512×512×1 and a fingerprint segmentation feature map of 64×64×1 respectively. The inputs and outputs of the phase filter set module are a normalized fingerprint image of 512×512×1 and a fingerprint phase feature map of 512×512×90 respectively; Subsequently, the complex image phase and direction field fusion feature map of the enhanced fingerprint module, the complex image amplitude and direction field fusion feature map inputs and the enhanced fingerprint output sizes are 512×512×90, 512×512×90 and 512×512×1 respectively.

[0116] ③ Perform concate and merge operations on the fingerprint segmentation feature map and the enhanced feature map. The input and output feature map sizes of this process are (512×512×1, 512×512×1) and 512×512×2 respectively.

[0117] ④ Pass the fused fingerprint feature map through the minutiae feature extraction module composed of ResNet to predict the final minutiae information. The input of this step is a fingerprint structure feature map of 512×512×2, and the outputs are a minutiae direction feature map of 64×64×180, a fingerprint H and W probability coordinate map of size 64×64×8, and a minutiae confidence score map of size 64×64×1.

[0118] ⑤ Calculate the target loss function L (using the calculation mode of cross-entropy and softmax regression) with the original labeled ground truth label and use the backpropagation algorithm for optimization iteration to obtain the optimal model.

[0119] Specifically, the module M5 adopts: applying non-maximum suppression based on Generalized Interaction of Union (GIoU) to delete subsequent fingerprint redundant points;

[0120] The calculation formula of the Generalized Interaction of Union is as follows:

[0121] Assume there are two arbitrary shapes E and F, find a smallest closed shape G that contains E and F, then calculate the proportion of the area in G that does not cover E and F to the area of G, and finally subtract the proportion of the area in G that does not cover E and F to the area of G from the IoU of E and F:

[0122]

[0123] Among them, As an evaluation index for whether it is an outlier.

[0124] In the subsequent processing steps of the minutiae points, for the first time in the present invention, a GIoU-based evaluation metric is used to perform redundant deletion on the minutiae points inferred by the deep learning model, obtaining a more complete fingerprint minutiae point output.

[0125] Example 2

[0126] Example 2 is a preferred example of Example 1

[0127] The present invention provides a fingerprint minutiae point extraction method based on ResNet feature extraction and Generalized Intersection over Union (GIoU) non-maximum suppression redundant deletion. Based on the existing prior knowledge and the neural network fingerprint minutiae point feature extraction architecture, this method proposes a more effective fingerprint minutiae point feature extraction algorithm. Since the neural network fingerprint detail feature extraction method based on the general CNN or VGG structure can no longer fully meet the actual needs in terms of recognition accuracy, a fingerprint feature extraction algorithm based on ResNet is developed to achieve fingerprint minutiae point detection. Subsequently, a non-maximum suppression method combined with the Generalized Intersection over Union metric is used to delete redundant minutiae points, further improving the accuracy of fingerprint minutiae point detection.

[0128] The present invention provides a fully automatic fingerprint minutiae point detection method based on deep learning, as Figure 1 described. This method is divided into training, testing, and deployment steps. Specifically, the method sequentially performs the following operations:

[0129] Step 1: Based on fingerprint prior knowledge and a fingerprint minutiae point prediction network of a convolutional neural network (the backbone structure is direction prediction and fingerprint segmentation based on ResNet, as well as minutiae point extraction), a fingerprint minutiae point extraction network D is formed.

[0130] Step 2: Train the constructed fingerprint minutiae point extraction network D. For each unit F=(I) in the training set, it passes through the fingerprint minutiae point extraction network D to generate a predicted minutiae point sequence P=(M p ), and the target loss function L is calculated by the prior attribute values of the original labeled ground truth minutiae points G=(M g ) and backpropagated for optimization iteration to obtain the optimal model. Here, I represents the original fingerprint image in the training set, M p represents the set of predicted minutiae points, and M g represents the set of ground truth minutiae points corresponding to the labeled fingerprint image.

[0131] Step 3: Perform inference and application on the optimal model obtained in Step 2 on the prediction dataset to initially predict the fingerprint minutiae point set, denoted as S naive ;

[0132] Step 4: Apply non-maximum suppression based on the general intersection over union to the initially obtained fingerprint minutiae points to remove subsequent fingerprint redundant points, and obtain an accurate fingerprint minutiae point set S final 。

[0133] Specifically, Step 1 adopts: constructing a model based on fingerprint prior knowledge and a deep convolutional neural network, and this method utilizes its feature representation ability to achieve end-to-end fingerprint image processing and minutiae detection.

[0134] Optionally, the construction principle of the fingerprint minutiae prediction network is based on the FingerNet architecture;

[0135] The method first uses a convolutional network with fixed weights to equivalently perform fingerprint image processing and minutiae extraction operations, specifically including fingerprint normalization, orientation field estimation, fingerprint segmentation, Gabor image enhancement, and minutiae extraction; subsequently, the simple network is expanded using the CNN or ResNet structure. In particular, the present invention uses the ResNet network to expand the key structure, greatly improving the ability to extract fingerprint features, so that the finally obtained deep neural network can be efficiently used for fingerprint minutiae detection after sufficient training. Based on FingerNet, the present invention redesigned the fingerprint orientation estimation and segmentation modules; in order to improve the accuracy and integrity of minutiae improvement, the present invention also iteratively updated and enhanced the minutiae extraction module.

[0136] The theoretical basis of the fingerprint orientation estimation module is gradient-based fingerprint orientation estimation. Convolution operations are used to replace gradient calculation and windowing operations. Here, the Sobel gradient operator (S x ,S y ) is used to perform the following orientation estimation operations:

[0137]

[0138] where I represents the input image of this module, S x represents the Sobel gradient operator along the x direction, I x represents the gradient map of the image along the x direction, I y represents the gradient map along the y direction, S y represents the Sobel gradient operator along the y direction, I xx represents w′w I x the sum of squared gradients, I yy represents w′w I y the sum of squared gradients, represents the ridge direction at this point, I xy represents w′w I x and I y the product sum, where x can represent the horizontal direction here and y can represent the vertical direction here.

[0139] In the above formula, the calculation method of atan2 is as follows:

[0140]

[0141] where J w is a matrix of all 1s with size w×w;

[0142] For the fingerprint image segmentation method, here convolution operation is used to replace the calculation operation of the local correlation degree (variance) of the image

[0143]

[0144] where Var represents the variance within the w×w square of the image, b represents the bias of the linear classifier here, which is similar to the threshold set during the foreground and background separation in the image processing algorithm.

[0145] Here, w is the length of the local window, k and b are the corresponding classifier parameters, [,, ] is the vector concatenation operation, and the above two operations can be equivalently completed by convolution operations with fixed parameters. Stacking these convolutions in an appropriate way can form the basic architecture of this module.

[0146] Optionally, as Figure 3 shown, in the fingerprint minutiae extraction module, the actual operation of the fingerprint minutiae detection defined based on minutiae is as follows: First, use the erosion operator on the enhanced fingerprint image to extract the fingerprint ridge skeleton information; then, according to the definition of the fingerprint ridge minutiae, locate the minutiae position information. The specific implementation approach is: traverse each pixel P in the fingerprint image, count the number of pixels different from P in the local block as count. If count = 5, then P is a crossing point; if count = 7, then P is an endpoint; otherwise, it is not a minutiae feature point. The equivalent convolution operation of the above process can be formalized as: Here I seg is the feature map after the fusion of fingerprint enhancement and segmentation, M t The convolution kernel parameters are jointly determined by the erosion operator and the definition operator of minutiae, and a maximum selection output (i.e., a maximization layer) operation is added after the convolution operation as the output of the basic architecture of this module.

[0147] Optionally, when training the fingerprint minutiae extraction network D, the open-source fingerprint image set NIST SD4 and its labeled minutiae information can be used as the training data set. The training data set of the proposed method is not limited to NIST SD4 and can also be other fingerprint data sets.

[0148] Optionally, in each training, the input and output of each module include: First, for the input fingerprint 3×3 image, the normalization operation directly adopts the per-pixel normalization method of a general network to adjust each pixel to a fixed ratio and range:

[0149]

[0150] where I(i,j) is the image intensity of the input image I at the pixel (i,j), Mean and Var are the image mean and variance respectively, M0 and Var0 are the image mean and variance after normalization respectively. The normalization operation here can be equivalent to the non-linear activation layer in a general network and can be implemented by an algorithm without parameter update during the training process. The input and output of this layer are a fingerprint image of 512×512×1 and a normalized fingerprint feature map respectively.

[0151] Optionally, the normalized feature map is input into the fingerprint orientation estimation and fingerprint image segmentation modules composed of ResNet in two branches respectively, and the other branch is input into the phase filter set module composed of 2D convolution, and then a filter in a suitable direction is selected for fingerprint image enhancement. The input and output of the orientation estimation module are a normalized feature map of 512×512×1 and an orientation feature map of 64×64×90 respectively; the input and output of the fingerprint image segmentation module are a normalized feature map of 512×512×1 and a fingerprint segmentation feature map of 64×64×1 respectively. As Figure 2As shown in the figure, the input normalized fingerprint feature map first enters two neural network layers with 64 3×3 convolutions in sequence, and then enters a 2×2 pooling layer with a stride of 2; then the feature map enters 128 3×3 convolutions and continues to enter two 128 3×3 convolutional layers. The latter two convolutional layers are connected by residuals, and this operation is repeated 3 times; then it enters a 2×2 pooling layer with a stride of 2, and the output of this step is 128 128×128 feature maps; the feature map continues to enter three neural network layers with 256 3×3 convolutional layers, and the latter two networks are connected by residuals, and then enters a 2×2 pooling layer with a stride of 2. Subsequently, the network backbone is divided into three branches. The first branch is the backbone branch, and the latter two branches are multi-scale segmentation networks based on convolutional kernel dilation operations to extract features within a larger receptive field, thereby improving the accuracy of fingerprint segmentation. In the backbone branch, the feature map enters three 512 3×3 convolutional layers, and the latter two layers are connected by residuals; the feature map enters two 256 3×3 convolutional layers, then enters a 128 1×1 convolutional layer, and then enters 90 1×1 direction estimation convolutional layers and one 1×1 segmentation convolutional layer respectively. In the latter two branches, the 128 128×128 feature maps output by the backbone network enter two convolutional networks with 256 3×3 convolutions with dilation rates of 4 and 8 respectively (used to improve the accuracy of fingerprint segmentation). The subsequent parts of these two convolutional networks have the same structure. First, they enter a 256 3×3 convolutional layer, then enter 128 1×1 convolutional layers respectively, and then enter 90 1×1 direction estimation convolutional layers and one 1×1 segmentation convolutional layer respectively. Finally, the direction features in the above three branches are fused and then sent to the activation layer (using Sigmoid as the activation function here) as the output of the direction features. Similarly, the fingerprint segmentation features are also fused and activated accordingly to output the segmentation feature map. In another branch, the input and output of the phase filter set module are the 512×512×1 normalized fingerprint image and the 512×512×90 fingerprint phase feature map respectively; then when enhancing the fingerprint image, the input and output feature map sizes are 512×512×90 and 512×512×1 respectively;

[0152] Optionally, when using Gabor for image enhancement, first use the complex Gabor filter g w,θ (generated by the local ridge frequency w and the local ridge direction θ) to perform convolution operations on the local fingerprint block I D . The enhanced complex block E D Each pixel can be represented as Here, A(x, y) and iφ(x, y) are the amplitude and phase of the enhanced complex block, and φ(x0, y0) is the final enhancement result. Since the Gabor filter does not share the weights of the entire image but only shares image patches with the same w and θ, to solve this problem, a selective filtering method is used here, that is, phase grouping is first performed followed by direction selection. First, phase grouping discretizes the parameters into N different intervals and generates Gabor filters respectively. A set of filtered complex images is obtained by performing a convolution operation using the Gabor filter and these images, which is expressed as follows: Among them, C(x, y, i) represents the intensity value of the complex image filtered for the i-th time at (x, y). The grouped phase F is the parameter of the grouped filtered image group; subsequently, a direction selection operation is performed. By generating a mask to select appropriate enhanced patches from the grouped phase, the definition of the i-th value at the pixel (x, y) in the mask M is: Finally, the enhancement map is calculated as: Based on the above theory, when designing the network, the direction estimated from the complex image and the phase filter map will directly generate a direction distribution map, which can be directly used as a direction mask to multiply the grouped phase.

[0153] Optionally, the finally feature point prediction is performed on the fused fingerprint feature map through a minutiae feature extraction module composed of ResNet. The input of this step is a fingerprint structure feature map of 512×512×2, and the outputs are a minutiae direction feature map of 64×64×180, a fingerprint H and W probability coordinate map of 64×64×8, and a minutiae confidence score map of 64×64×1. The specific structure design of this step is as follows: the 512×512×2 feature map first enters a 3-layer convolutional network layer, and each layer contains 64 9×9 convolutions. The latter two convolutional layers are connected through a residual network. The above operations are looped 3 times, and then enter a Pooling layer with a stride of 2 and a pooling size of 2×2; then, the 256×256×128 feature map enters a 3-layer convolutional network, and each layer of the network includes 128 5×5 convolutions. Similarly, the latter two layers are connected by residuals. This step is looped 3 times and then enters a pooling layer with a stride of 2 and a pooling size of 2×2; then enters a 3-layer convolutional network, and each layer of the network includes 256 3×3 convolutions. The latter two convolutional layers are connected by residuals. Finally, the network enters a pooling layer with a stride of 2 and a pooling size of 2×2; next, the 64×64×256 feature map will enter 3 branches to evaluate the probabilities of direction, minutiae coordinates (h, w), and confidence scores respectively: in the direction evaluation branch, first fuse and merge the minutiae features and the direction output features, and then send them into 256 1×1 convolutional blocks and 180 1×1 convolutional layers in sequence, and then enter the sigmoid layer to finally obtain the direction probability distribution map of the minutiae; for the minutiae position (h, w), it will enter 256 1×1 convolutional layers and 8 1×1 convolutional layers in sequence, and then enter the sigmoid layer to obtain the minutiae position probability distribution map; for the minutiae confidence score value (score), it will enter 256 1×1 convolutional layers and 1 1×1 convolutional layer in sequence, and then enter the sigmoid layer to obtain the minutiae confidence score distribution map.

[0154] Optionally, apply non-maximum suppression based on Generalized Intersection over Union (GIoU) to perform subsequent fingerprint redundant point elimination. The calculation method of GIoU here is as follows: Figure 4 As shown, assuming there are two arbitrary shapes E and F, we find a smallest closed shape G that contains E and F. Then calculate the proportion of the area in G that does not cover E and F to the area of G. Finally, subtract this ratio from the IoU of E and F: As an evaluation index for whether it is an abnormal point; in this step, for the first time, the present invention uses the evaluation index based on GIoU to perform redundant deletion on the detail points initially inferred by the model, and obtains the final accurate fingerprint detail point output; the specific steps of this module include: first, arrange the detected detail points into a sequence order according to the credibility level; then mark the point with the highest detail point score as saved, and calculate the GIoU value between this point and the remaining points in order in turn. If the GIoU value is greater than thresh, then this detail point is deleted, otherwise it is retained in order, and the information of all points in order is updated according to this operation; re-sort order according to the detail point credibility information, and repeat the above operation in a loop until the detail points in the order set are empty. The marked saved points are used as the final output to complete the redundant point deletion operation.

[0155] Those skilled in the art know that in addition to implementing the systems, devices and their respective modules provided by the present invention in the form of pure computer-readable program codes, the method steps can be logically programmed to enable the systems, devices and their respective modules provided by the present invention to be implemented in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, embedded microcontrollers, etc. to implement the same program. Therefore, the systems, devices and their respective modules provided by the present invention can be regarded as a kind of hardware component, and the modules included therein for implementing various programs can also be regarded as the structures within the hardware component; the modules for implementing various functions can also be regarded as either software programs for implementing the method or the structures within the hardware component.

[0156] The specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various changes or modifications within the scope of the claims, which does not affect the essence of the present invention. Without conflict, the embodiments of the present application and the features in the embodiments can be combined with each other arbitrarily.

Claims

1. A fully automatic fingerprint minutiae extraction method, characterized in that, Including: Step S1: Based on fingerprint prior knowledge and a fingerprint minutiae preliminary prediction network, a fingerprint minutiae extraction network D is formed. Step S2: Preprocess the fingerprint image to obtain the preprocessed fingerprint image. Step S3: Use the preprocessed fingerprint image to train the fingerprint minutiae extraction network D to obtain the trained fingerprint minutiae extraction network D. Step S4: Use the trained fingerprint minutiae extraction network D to preliminarily predict the fingerprint minutiae point set. Step S5: Apply the non-maximum suppression algorithm based on the general intersection over union to the preliminarily predicted fingerprint minutiae point set to remove redundant fingerprint points and obtain the final accurate fingerprint minutiae point set. The fingerprint minutiae preliminary prediction network is a convolutional neural network for direction prediction, fingerprint segmentation, and minutiae extraction based on ResNet. The fingerprint minutiae extraction network D is a minutiae detection method based on the FingerNet framework, including using a residual network to predict the fingerprint direction field for fingerprint image segmentation and using a residual network to predict fingerprint minutiae points, and finally deleting redundant points based on the proposed general intersection over union non-maximum suppression algorithm. The fingerprint direction estimation and fingerprint image segmentation module composed of ResNet includes: The input normalized fingerprint feature map first enters two neural network layers containing 64 3×3 convolutions in sequence, and then enters a 2×2 pooling layer with a stride of 2. Then the feature map enters 128 3×3 convolutions and continues to enter two 128 3×3 convolutional layers. The latter two convolutional layers are connected by residuals, and this operation is repeated 3 times. After that, it enters a 2×2 pooling layer with a stride of 2, and the output of this step is 128 128×128 feature maps. The feature map continues to enter three neural network layers of 256 3×3 convolutional layers, and the latter two networks are connected by residuals. Then it enters a 2×2 pooling layer with a stride of 2. Subsequently, the network backbone is divided into three branches. The first branch is the main branch, and the latter two branches are multi-scale segmentation networks based on convolutional kernel dilation operations to extract features within a larger receptive field, thereby improving the accuracy of fingerprint segmentation. In the main branch, the feature map enters three 512 3×3 convolutional layers, and the latter two layers are connected by residuals. The feature map enters two 256 3×3 convolutional layers, then enters a 128 1×1 convolutional layer, and then enters a 90 1×1 direction estimation convolutional layer and a 1×1 segmentation convolutional layer respectively. In the latter two branches, the 128 128×128 feature maps output by the backbone network enter two convolutional networks of 256 3×3 with dilation rates of 4 and 8 respectively. The subsequent parts of these two convolutional networks have the same structure. First, they enter a 256 3×3 convolutional layer, then enter a 128 1×1 convolutional layer respectively, and then enter a 90 1×1 direction estimation convolutional layer and a 1×1 segmentation convolutional layer respectively. Finally, the direction features in the above three branches are fused and then sent to the activation layer as the output of the direction features. Similarly, the fingerprint segmentation features are also fused and activated accordingly to output the segmentation feature map. The specific structural design of the detail point feature extraction module composed of ResNet is as follows: The 512×512×2 feature map first enters a 3-layer convolutional network layer. Each layer contains 64 9×9 convolutions. The latter two convolutional layers are connected through a residual network. The above operations are looped 3 times, and then it enters a Pooling layer with a stride of 2 and a pooling size of 2×2; afterwards, the 256×256×128 feature map enters a 3-layer convolutional network. Each layer of the network includes 128 5×5 convolutions. Similarly, the latter two layers are connected by residuals. This step is looped 3 times and then enters a pooling layer with a stride of 2 and a pooling size of 2×2; then it enters a 3-layer convolutional network. Each layer of the network includes 256 3×3 convolutions. The latter two convolutional layers are connected by residuals. Finally, the network enters a pooling layer with a stride of 2 and a pooling size of 2×2; next, the 64×64×256 feature map will enter 3 branches to respectively evaluate the probability values of the direction, the detail point coordinates (h, w), and the confidence score.

2. The fully automatic fingerprint minutiae extraction method according to claim 1, characterized in that, The step S2 adopts: Adopt a per-pixel normalization method for the fingerprint image to adjust each pixel to a fixed ratio and range; Among them, I(i, j) represents the image intensity of the input image I at the pixel (i, j); Mean and Var respectively represent the image mean and variance; M0 and Var0 respectively represent the image mean and variance after normalization.

3. The fully automatic fingerprint minutiae extraction method according to claim 1, characterized in that The step S3 adopts: the preprocessed fingerprint image passes through the fingerprint minutiae extraction network D to generate a predicted minutiae point sequence P = (M p ), and calculates the attribute information difference loss function L with the original labeled ground truth minutiae G = (M g ), and backpropagates for optimization iteration to obtain the trained fingerprint minutiae extraction network D; where M p represents the predicted minutiae point set, and M g represents the ground truth minutiae point set labeled for the corresponding fingerprint image.

4. The fully automatic fingerprint minutiae extraction method according to claim 1, characterized in that The fingerprint detail extraction network D includes: a fingerprint direction estimation and fingerprint image segmentation module, a set of phase filter modules, an enhanced fingerprint module, and a ResNet detail point extraction module composed of ResNet; Step S3.1: Pass the preprocessed fingerprint image through a set of phase filter modules composed of 2D convolutions to obtain a fingerprint phase feature map; Step S3.2: Pass the original fingerprint image through a set of phase filter modules composed of 2D convolutions to obtain an amplitude feature map; Step S3.3: Pass the preprocessed fingerprint image through the fingerprint direction estimation and fingerprint image segmentation module composed of ResNet to obtain a fingerprint direction feature map and a fingerprint segmentation feature map; Step S3.4: Pass the obtained fingerprint phase feature map, fingerprint direction feature map, and amplitude feature map through the enhanced fingerprint module for fingerprint enhancement to obtain an enhanced feature map; Step S3.5: Perform concate and merge operations on the fingerprint segmentation feature map and the enhanced feature map to obtain a fused fingerprint feature map; Step S3.6: Pass the fused fingerprint feature map through the ResNet detail point extraction module to predict the final feature point information and generate predicted detail points; Step S3.7: Calculate the target loss function L between the predicted detail points and the original labeled true detail points, and use the backpropagation algorithm to optimize the target loss L. Repeat triggering steps S3.1 to S3.7 until the objective function converges to obtain the trained fingerprint detail extraction network D.

5. The fully automatic fingerprint minutiae extraction method according to claim 1, characterized in that, The step S5 adopts: The calculation formula of the general intersection over union is as follows: Suppose there are two arbitrarily shaped objects E and F. Find the smallest closed shape G that encloses both E and F. Then calculate the proportion of the area of G that does not cover E and F. Finally, subtract this proportion from the IoU of E and F: Among them, is used as an evaluation index for whether it is an abnormal point.

6. A fully automatic fingerprint minutiae extraction system, characterized in that, Including: Module M1: Based on fingerprint prior knowledge and a fingerprint minutiae preliminary prediction network, form a fingerprint minutiae extraction network D; Module M2: Preprocess the fingerprint image to obtain the preprocessed fingerprint image; Module M3: Use the preprocessed fingerprint image to train the fingerprint minutiae extraction network D to obtain the trained fingerprint minutiae extraction network D; Module M4: Use the trained fingerprint minutiae extraction network D to preliminarily predict a set of fingerprint minutiae; Module M5: Apply a non-maximum suppression algorithm based on the universal intersection over union to the preliminarily predicted set of fingerprint minutiae to remove redundant fingerprint points and obtain the final accurate set of fingerprint minutiae; The fingerprint minutiae preliminary prediction network is a convolutional neural network for direction prediction, fingerprint segmentation, and minutiae extraction based on ResNet; The fingerprint minutiae extraction network D is a minutiae detection method based on the FingerNet framework; it includes using a residual network to predict the fingerprint orientation field for fingerprint image segmentation and using a residual network to predict fingerprint minutiae, and finally deleting redundant points based on the proposed universal intersection over union non-maximum suppression algorithm; The fingerprint orientation estimation and fingerprint image segmentation module composed of ResNet includes: The input normalized fingerprint feature map first enters two neural network layers with 64 3×3 convolutions in sequence, and then enters a 2×2 pooling layer with a stride of 2; then the feature map enters 128 3×3 convolutions and continues to enter two 128 3×3 convolutional layers. The latter two convolutional layers are connected by residuals, and this operation is repeated 3 times; then it enters a 2×2 pooling layer with a stride of 2, and the output of this step is 128 feature maps of 128×128; the feature map continues to enter a neural network with three 256 3×3 convolutional layers, and the latter two networks are connected by residuals, and then enters a 2×2 pooling layer with a stride of 2; subsequently, the network backbone is divided into three branches. The first branch is the backbone branch, and the latter two branches are multi-scale segmentation networks based on convolutional kernel dilation operations to extract features within a larger receptive field, thereby improving the accuracy of fingerprint segmentation; in the backbone branch, the feature map enters three 512 3×3 convolutional layers, and the latter two layers are connected by residuals; the feature map enters two 256 3×3 convolutional layers, then enters 128 1×1 convolutional layers, and then enters 90 1×1 direction estimation convolutional layers and 1 1×1 segmentation convolutional layer respectively; in the latter two branches, the 128 128×128 feature maps output by the backbone network enter two convolutional networks with 256 3×3 convolutions with dilation rates of 4 and 8 respectively. The subsequent parts of these two convolutional networks have the same structure. First, they enter a 256 3×3 convolutional layer, then enter 128 1×1 convolutional layers respectively, and then enter 90 1×1 direction estimation convolutional layers and 1 1×1 segmentation convolutional layer respectively; finally, the direction features in the above three branches are fused and then sent to the activation layer as the output of the direction features. Similarly, the fingerprint segmentation features are also fused and activated accordingly to output the segmentation feature map; The specific structural design of the minutiae feature extraction module composed of ResNet is as follows: The 512×512×2 feature map first enters three convolutional network layers, each layer contains 64 9×9 convolutions, and the latter two convolutional layers are connected by the residual network. The above operation is repeated 3 times, and then enters a Pooling layer with a stride of 2 and a pooling size of 2×2; then, the 256×256×128 feature map enters three convolutional networks, each layer of the network includes 128 5×5 convolutions, and the latter two layers also make residual connections. This step of operation is repeated 3 times and enters a pooling layer with a stride of 2 and a pooling size of 2×2; then it enters three convolutional networks, each layer of the network includes 256 3×3 convolutions, and the latter two convolutional layers are connected by residuals. Finally, the network enters a pooling layer with a stride of 2 and a pooling size of 2×2; next, the 64×64×256 feature map will enter three branches to evaluate the probability values of the direction, minutiae coordinates (h, w), and confidence scores respectively.

7. The fully automatic fingerprint minutiae extraction system according to claim 6, characterized in that, The module M2 adopts: For the fingerprint image, a per-pixel normalization method is adopted to adjust each pixel to a fixed ratio and interval; Among them, I(i, j) represents the image intensity of the input image I at the pixel (i, j); Mean and Var represent the image mean and variance respectively; M0 and Var0 represent the normalized image mean and variance respectively.

8. The fully automatic fingerprint minutiae extraction system according to claim 6, characterized in that, The module M3 adopts the following: the preprocessed fingerprint image passes through the fingerprint minutiae extraction network D to generate a predicted minutiae point sequence P = (M p ), and calculates the attribute information difference loss function L with the original labeled ground truth minutiae G = (M g ), and backpropagates for optimization iteration to obtain the trained fingerprint minutiae extraction network D; where M p represents the predicted minutiae point set, and M g represents the ground truth minutiae point set labeled for the corresponding fingerprint image.

9. The fully automatic fingerprint minutiae extraction system according to claim 6, wherein The fingerprint minutiae extraction network D includes: a fingerprint orientation estimation and fingerprint image segmentation module, a phase filter set module, an enhanced fingerprint module, and a ResNet minutiae extraction module, which are composed of ResNet; Module M3.1: Obtain a fingerprint phase feature map by passing the preprocessed fingerprint image through a phase filter set module composed of 2D convolutions; Module M3.2: Obtain an amplitude feature map by passing the original fingerprint image through a phase filter set module composed of 2D convolutions; Module M3.3: Pass the preprocessed fingerprint image through a fingerprint orientation estimation and fingerprint image segmentation module composed of ResNet to obtain a fingerprint orientation feature map and a fingerprint segmentation feature map; Module M3.4: Enhance the fingerprint by passing the obtained fingerprint phase feature map, fingerprint orientation feature map, and amplitude feature map through an enhanced fingerprint module to obtain an enhanced feature map; Module M3.5: Perform concatenation and merging operations on the fingerprint segmentation feature map and the enhanced feature map to obtain a fused fingerprint feature map; Module M3.6: Perform final feature point information prediction on the fused fingerprint feature map through a ResNet minutiae extraction module to generate predicted minutiae; Module M3.7: Calculate the objective loss function L between the predicted minutiae and the original labeled ground truth minutiae, and use the backpropagation algorithm to optimize the objective loss L. Repeat triggering Module M3.1 to Module M3.7 until the objective function converges to obtain the trained fingerprint minutiae extraction network D.

10. The fully automatic fingerprint minutiae extraction system according to claim 6, characterized in that The module M5 adopts: The calculation formula of the general intersection over union is as follows: Assume there are two arbitrary shapes E and F. Find a smallest closed shape G such that G contains E and F. Then calculate the proportion of the area in G that does not cover E and F to the area of G. Finally, subtract the proportion of the area in G that does not cover E and F to the area of G from the IoU of E and F: Among them, is used as an evaluation index for whether it is an abnormal point.

Images