Non-contact fingerprint image correction method and device, electronic equipment and storage medium

Abstract

The invention provides a non-contact fingerprint image correction method and device, electronic equipment and a storage medium, and belongs to the technical field of artificial intelligence, and the method comprises the steps: obtaining an original image of an uncorrected non-contact fingerprint, inputting the original image into a fingerprint image correction model, and obtaining a first pixel remapping coordinate; performing remapping processing on the original image based on the first pixel remapping coordinates to obtain a corrected target image of the non-contact fingerprint; the fingerprint image correction model is obtained after an image decoder branch and an image generation module of the fingerprint image correction network are cut; the fingerprint image correction network is obtained by pre-training a plurality of training samples; each training sample comprises a non-contact fingerprint image sample and a remapping coordinate label and a correction image label corresponding to the non-contact fingerprint image sample. According to the invention, on the basis of reducing the hardware requirement, difficulty and cost of the fingerprint image acquisition equipment, the correction of the perspective deformation of the non-contact fingerprint image can be well realized without depth estimation and three-dimensional reconstruction.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular, to a method, device, electronic device and storage medium for correcting non-contact fingerprint images. Background Art

[0002] Non-contact fingerprint recognition technology is a type of fingerprint recognition technology. When performing non-contact fingerprint recognition, the surface curvature of the finger will cause obvious perspective distortion in the captured finger image, resulting in changes in the positions of fingerprint ridges, periods, minutiae, feature points, etc. Therefore, before recognizing non-contact fingerprints, it is necessary to correct the collected non-contact fingerprint images.

[0003] Currently, there are two common methods for correcting the collected non-contact fingerprint images: one is the correction method based on texture analysis, that is, extracting and analyzing the texture of the obtained non-contact fingerprint image and performing correction according to the texture analysis results; the other is the correction method based on 3D modeling, that is, after performing depth estimation and three-dimensional reconstruction based on the gradient of a single obtained non-contact fingerprint image, then correcting the non-contact fingerprint image.

[0004] However, both the correction method based on texture analysis and the correction method based on 3D modeling are greatly affected by the non-contact fingerprint image capture device and the capture environment. If the non-contact fingerprint image capture device and the capture environment cannot meet the professional requirements, on the one hand, it will cause the fingerprint extraction and analysis results of the non-contact fingerprint image to not meet the correction requirements; on the other hand, it will cause the non-contact fingerprint image to have a large amount of noise and inaccurate image gradient information, resulting in more computational resources and time being consumed for depth estimation and three-dimensional reconstruction, and even possibly ultimately causing the correction result of the non-contact fingerprint image to not meet the actual usage requirements.

[0005] That is to say, if the correction method based on texture analysis and the correction method based on 3D modeling are to achieve non-contact fingerprint image correction, it is necessary to use a more expensive professional image capture device and a higher-performance computing device to provide more computational resources and time, resulting in low applicability of the image correction method.

[0006] Therefore, it is necessary to provide a non-contact fingerprint image correction solution that does not rely on professional image capture devices and the performance of computing devices and has stronger applicability. Summary of the Invention

[0007] The present invention provides a method, device, electronic device and storage medium for correcting non-contact fingerprint images, so as to solve the defect in the prior art that the high cost of professional image capture devices and the high performance requirements of computing devices lead to low applicability of the image correction method, and to implement a non-contact fingerprint image correction solution that does not rely on professional image capture devices and the performance of computing devices and has stronger applicability.

[0008] The present invention provides a method for correcting the image of a non-contact fingerprint, including: Obtaining an original image of an uncorrected non-contact fingerprint; Inputting the original image into a fingerprint image correction model to obtain first pixel remapping coordinates output by the fingerprint image correction model; Performing remapping processing on the original image based on the first pixel remapping coordinates to obtain a target image of the corrected non-contact fingerprint; The fingerprint image correction model is obtained after cropping the image decoder branch and the image generation module in a fingerprint image correction network; the fingerprint image correction network is pre-trained based on a plurality of training samples; each training sample includes a non-contact fingerprint image sample and its corresponding remapping coordinate label and corrected image label.

[0009] According to the method for correcting the image of a non-contact fingerprint provided by the present invention, the fingerprint image correction network includes a feature extraction module, an encoder module, a decoder module, a coordinate remapping module, and the image generation module; the decoder module includes a coordinate decoder branch and the image decoder branch; The feature extraction module is configured to determine fingerprint extraction features based on the non-contact fingerprint image sample; The encoder module is configured to determine multi-scale encoder features based on the fingerprint extraction features; The coordinate decoder branch is configured to determine coordinate remapping features based on the multi-scale encoder features; The image decoder branch is configured to determine multi-scale image features based on the multi-scale encoder features; The coordinate remapping module is configured to determine second pixel remapping coordinates based on the coordinate remapping features; The image generation module is configured to determine a fingerprint corrected image based on the multi-scale image features and the fingerprint extraction features.

[0010] According to the method for correcting the image of a non-contact fingerprint provided by the present invention, the Transformer structures in the encoder module and the decoder module are constructed based on a multi-head attention mechanism.

[0011] According to the method for correcting the image of a non-contact fingerprint provided by the present invention, the encoder module includes a first-layer encoder, a first-layer first convolutional layer, a second-layer encoder, a second-layer first convolutional layer, and a third-layer encoder connected in sequence; The first-layer encoder is configured to determine first-scale encoder features based on the fingerprint extraction features; The first convolutional layer of the first layer is used to downsample the output features of the encoder of the first layer; The encoder of the second layer is used to determine the encoder features of the second scale based on the output features of the first convolutional layer of the first layer; The first convolutional layer of the second layer is used to downsample the output features of the encoder of the second layer; The encoder of the third layer is used to determine the encoder features of the third scale based on the output features of the first convolutional layer of the first layer.

[0012] According to an image correction method for non-contact fingerprints provided by the present invention, the coordinate decoder branch includes a first-layer coordinate decoder, a first second convolutional layer, a second-layer coordinate decoder, a second second convolutional layer, and a third-layer coordinate decoder connected in sequence; The input of the first-layer coordinate decoder is determined based on the encoder features of the first scale; The input of the second-layer coordinate decoder is determined based on the encoder features of the second scale and the output features of the first second convolutional layer; The input of the third-layer coordinate decoder is determined based on the encoder features of the third scale and the output features of the second second convolutional layer.

[0013] According to an image correction method for non-contact fingerprints provided by the present invention, the coordinate remapping module includes a remapped coordinate initialization layer, a third convolutional layer, a remapped coordinate fusion layer, and a remapped coordinate upsampling layer; The remapped coordinate initialization layer is used to fixedly initialize the original value of the coordinate remapping feature; The third convolutional layer is used to determine the initially fitted initial remapped coordinates based on the coordinate remapping feature; The remapped coordinate fusion layer is used to fuse the initial remapped coordinates and the original value to obtain the fused remapped coordinates; The remapped coordinate upsampling layer is used to upsample the fused remapped coordinates in combination with the coordinate remapping feature to determine the second pixel remapped coordinates.

[0014] According to an image correction method for non-contact fingerprints provided by the present invention, the image decoder branch includes a first-layer image decoder, a first fourth convolutional layer, a second-layer image decoder, a second fourth convolutional layer, and a third-layer image decoder connected in sequence; The input of the first-layer image decoder is determined based on the encoder features of the first scale, and the output of the first-layer image decoder is the image features of the first scale; The input of the second-layer image decoder is determined based on the second-scale encoder features and the output features of the fourth convolutional layer of the first layer, and the output of the second-layer image decoder is the second-scale image features; The input of the third-layer image decoder is determined based on the third-scale encoder features and the output features of the fourth convolutional layer of the second layer, and the output of the third-layer image decoder is the third-scale image features.

[0015] According to an image correction method for non-contact fingerprints provided by the present invention, the image generation module includes a first-layer upsampling convolutional layer, a second-layer upsampling convolutional layer, a third-layer upsampling convolutional layer, a fourth-layer upsampling convolutional layer, and a fifth-layer upsampling convolutional layer connected in sequence; The input of the first-layer upsampling convolutional layer is determined based on the first-scale image features; The input of the second-layer upsampling convolutional layer is determined based on the output features of the first-layer upsampling convolutional layer and the second-scale image features; The input of the third-layer upsampling convolutional layer is determined based on the output features of the second-layer upsampling convolutional layer and the third-scale image features; The input of the fourth-layer upsampling convolutional layer is determined based on the output features of the third-layer upsampling convolutional layer and the fingerprint extraction features; The input of the fifth-layer upsampling convolutional layer is determined based on the output features of the fourth-layer upsampling convolutional layer and the fingerprint extraction features.

[0016] According to an image correction method for non-contact fingerprints provided by the present invention, the multiple training samples are obtained based on the following method: Based on the manual annotation method of remapped coordinates, the first quantity of the training samples is obtained; Based on the automatic generation method of remapped coordinates, the second quantity of the training samples is obtained; The first quantity is less than the second quantity.

[0017] According to an image correction method for non-contact fingerprints provided by the present invention, the step of obtaining the first quantity of the training samples based on the manual annotation method of remapped coordinates includes: Obtain the first quantity of uncorrected non-contact fingerprint images as the non-contact fingerprint image samples; Manually annotate the remapped coordinate points of the uncorrected non-contact fingerprint images to obtain the remapped coordinate labels corresponding to the non-contact fingerprint image samples; Based on the remapped coordinate labels, perform remapping processing on the uncorrected non-contact fingerprint images to obtain the corrected image labels corresponding to the non-contact fingerprint image samples; Determine the first quantity of the training samples based on the non-contact fingerprint image samples, the remapping coordinate labels, and the corrected image labels.

[0018] According to an image correction method for non-contact fingerprints provided by the present invention, the obtaining of the second quantity of the training samples based on the automatic generation method of remapping coordinates includes: Obtain a third quantity of approximately corrected contact fingerprint images as the corrected image labels; Automatically generate remapping coordinate points of the approximately corrected contact fingerprint images according to a preset fingerprint perspective deformation rule to obtain the second quantity of the remapping coordinate labels; Perform inverse remapping processing on the approximately corrected contact fingerprint images based on the remapping coordinate labels to obtain the second quantity of the non-contact fingerprint image samples; Determine the second quantity of the training samples based on the second quantity of the non-contact fingerprint image samples, the second quantity of the remapping coordinate labels, and the third quantity of the corrected image labels; The third quantity is less than the second quantity.

[0019] The present invention also provides an image correction device for non-contact fingerprints, including: An uncorrected image acquisition module, configured to acquire an original image of an uncorrected non-contact fingerprint; A remapping coordinate acquisition module, configured to input the original image into a fingerprint image correction model to obtain first pixel remapping coordinates output by the fingerprint image correction model; A fingerprint remapping processing module, configured to perform remapping processing on the original image based on the first pixel remapping coordinates to obtain a target image of the corrected non-contact fingerprint; The fingerprint image correction model is obtained by cropping an image decoder branch and an image generation module in a fingerprint image correction network; the fingerprint image correction network is pre-trained based on a plurality of training samples; each training sample includes a non-contact fingerprint image sample and its corresponding remapping coordinate label and corrected image label.

[0020] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and running on the processor, where when the processor executes the computer program, the image correction method for non-contact fingerprints as described in any one of the above is implemented.

[0021] The present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the image correction method for non-contact fingerprints as described in any one of the above is implemented.

[0022] The present invention also provides a computer program product, including a computer program which, when executed by a processor, implements the image correction method for non-contact fingerprint as described in any one of the above.

[0023] The image correction method, device, electronic device and storage medium for non-contact fingerprint provided by the present invention construct a fingerprint image correction network with an image decoder branch and an image generation module during model pre-training, and use training samples including non-contact fingerprint image samples and their corresponding remapping coordinate labels and corrected image labels to train the fingerprint image correction network, so as to utilize the auxiliary supervised training results of the image decoder branch and the image generation module to enable the network structure before the image decoder branch and the image generation module in the fingerprint image correction network to extract better image features. After the pre-training is completed, the image decoder branch and the image generation module are cropped to obtain a fingerprint image correction model for inference, which can better correct the perspective distortion of non-contact fingerprint images on the basis of reducing the hardware requirements, acquisition difficulty and acquisition cost of the fingerprint original image acquisition device, and at the same time, there is no need for depth estimation and three-dimensional reconstruction, greatly reducing the requirements for hardware computing resources, having stronger applicability and a wider application range, and being more conducive to improving the recognition accuracy of non-contact fingerprints. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present invention, and those of ordinary skill in the art can also obtain other drawings based on these drawings without creative efforts.

[0025] Figure 1 is a schematic flowchart of the image correction method for non-contact fingerprint provided by the present invention.

[0026] Figure 2 is a schematic structural diagram of the fingerprint image correction network provided by the present invention.

[0027] Figure 3 is a schematic structural diagram of the fingerprint image correction model provided by the present invention.

[0028] Figure 4 is a schematic structural diagram of the encoder module provided by the present invention.

[0029] Figure 5 is a schematic structural diagram of the decoder module provided by the present invention.

[0030] Figure 6 is a schematic structural diagram of the coordinate remapping module provided by the present invention.

[0031] Figure 7 It is a schematic structural diagram of the image generation module provided by the present invention.

[0032] Figure 8 It is an acquisition example diagram of the contact fingerprint image after approximate correction provided by the present invention.

[0033] Figure 9 It is a schematic structural diagram of the non-contact fingerprint image correction device provided by the present invention.

[0034] Figure 10 It is a schematic structural diagram of the electronic device provided by the present invention. Detailed implementation manners

[0035] To make the objectives, technical solutions and advantages of the present invention clearer, the technical solutions in the present invention will be clearly and completely described below with reference to the accompanying drawings in the present invention. Apparently, the described embodiments are some but not all of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art without creative efforts based on the embodiments in the present invention belong to the scope of protection of the present invention.

[0036] It should be noted that in the description of the present invention, the term "comprising" or any other variant thereof is intended to cover a non-exclusive inclusion, so that a process, method, article or device comprising a series of elements not only includes those elements but also includes other elements not expressly listed, or further includes elements inherent to such a process, method, article or device. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0037] The terms "first", "second", etc. in the present invention are used to distinguish similar objects and are not used to describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances so that the embodiments of the present invention can be implemented in an order different from those illustrated or described herein, and the objects distinguished by "first", "second", etc. generally belong to the same category, and the number of objects is not limited. For example, the first object can be one or more.

[0038] The following combines Figures 1-10 to describe the non-contact fingerprint image correction method, device, electronic device and storage medium provided by the present invention.

[0039] Fingerprint recognition technology is currently one of the widely used biometric recognition technologies, which is applied in all aspects of real life, such as unlocking electronic devices like mobile phones and tablets, access control systems such as door locks and safes, enterprise management fields such as employee attendance, financial fields such as fingerprint payment and bank services, and public service fields such as population management and immigration management.

[0040] As a type of fingerprint recognition technology, non-contact fingerprint recognition technology is an advanced biometric recognition technology. Compared with traditional contact fingerprint recognition technology, it has characteristics such as hygiene, portability, and strong adaptability. However, it also faces some limitations and challenges. The surface curvature of the non-contact fingerprint finger will cause obvious perspective distortion in the captured finger image, which will change the positions of fingerprint ridges, periods, minutiae points, feature points, etc., bringing difficulties to fingerprint recognition and resulting in poor fingerprint recognition accuracy. Therefore, in the non-contact fingerprint recognition scenario, before recognizing the non-contact fingerprint, it is necessary to correct the captured non-contact fingerprint image.

[0041] Different from the traditional contact fingerprint image correction which is to correct the fingerprint deformation distortion caused by different degrees of finger pressing during the fingerprint image acquisition process, non-contact fingerprint image correction is to correct the obvious perspective distortion in the captured image caused by the finger surface curvature. Currently, the commonly used non-contact fingerprint image correction methods include two methods: the correction method based on texture analysis and the correction method based on 3D modeling. If these two correction methods are to achieve non-contact fingerprint image correction, they must use more expensive professional image capture devices and computing devices with higher performance and cost to provide more computing resources and time, resulting in low applicability of the image correction method.

[0042] In view of this, the present invention provides an image correction method, device, electronic device, and storage medium for non-contact fingerprints to achieve a non-contact fingerprint image correction solution with stronger applicability without relying on professional image capture devices and computing device performance.

[0043] Figure 1 It is a schematic flowchart of the image correction method for non-contact fingerprints provided by the present invention. As Figure 1 shown, the image correction method for non-contact fingerprints includes but is not limited to steps 101 to 103.

[0044] It should be noted that the execution subject of the non-contact fingerprint image correction method provided by the present invention can be a server or a computer device, such as a mobile phone, a tablet computer, a laptop computer, a handheld computer, a vehicle-mounted electronic device, a wearable device, an Ultra-Mobile Personal Computer (UMPC), a netbook, or a Personal Digital Assistant (PDA), etc.

[0045] Step 101: Obtain the original image of the uncorrected non-contact fingerprint.

[0046] Specifically, an imaging device such as a mobile phone or a camera is used to capture an image of the uncorrected non-contact fingerprint as the original image.

[0047] It should be noted that for the imaging device that collects the original image of the uncorrected non-contact fingerprint, when the imaging device meets the necessary shooting clarity and resolution, the present invention does not impose additional restrictions on the image acquisition device, including neither requiring the non-contact fingerprint image captured by the imaging device to meet the texture requirements required by the correction method based on texture analysis, nor requiring the non-contact fingerprint image captured by the imaging device to meet the gradient requirements required by the correction method based on 3D modeling, etc.

[0048] Step 102: Input the original image into the fingerprint image correction model to obtain the first pixel remapping coordinates output by the fingerprint image correction model.

[0049] The fingerprint image correction model is obtained after cropping the image decoder branch and the image generation module in the fingerprint image correction network; the fingerprint image correction network is pre-trained based on multiple training samples; each training sample includes a non-contact fingerprint image sample, its corresponding remapping coordinate label, and a corrected image label.

[0050] The first pixel remapping coordinates are the coordinates of the remapping points obtained after the fingerprint image correction model processes the original image of the uncorrected non-contact fingerprint during the inference stage of the fingerprint image correction model, and are used to perform remapping processing on the pixels of the original image, so as to correct the original image with pixel perspective distortion caused by the curvature of the finger surface.

[0051] The image decoder branch and the image generation module are used to implement the auxiliary supervised training during the pre-training of the fingerprint image correction network, so that the network structure of the fingerprint image correction network located before the image decoder branch and the image generation module can extract better image features.

[0052] Specifically, before using the fingerprint image correction model for inference, a fingerprint image correction network is first constructed by combining Transformer and Convolutional Neural Networks (CNN). The constructed fingerprint image correction network includes an image processing branch and a coordinate processing branch, where the image processing branch includes an image decoder branch and an image generation module.

[0053] The fingerprint image correction network is trained using multiple training samples each consisting of a non-contact fingerprint image sample, its corresponding remapped coordinate label, and a corrected image label. The non-contact fingerprint image sample is input into the fingerprint image correction network to obtain an image output value output by the image processing branch of the fingerprint image correction network and a coordinate output value output by the coordinate processing branch. Then, the network parameters of the fingerprint image correction network are optimized using the image output value and the coordinate output value, that is, the network parameters of the fingerprint image correction network are optimized according to the error between the image output value and the corrected image label corresponding to the non-contact fingerprint image sample, and at the same time, the network parameters of the fingerprint image correction network are optimized according to the error between the coordinate output value and the remapped coordinate label corresponding to the non-contact fingerprint image sample.

[0054] The fingerprint image correction network is continuously iteratively trained using multiple training samples until termination conditions such as the iteration count reaching the maximum iteration count and the model output accuracy meeting the preset accuracy requirements are satisfied, completing the pre-training of the fingerprint image correction network.

[0055] The image decoder branch and the image generation module in the pre-trained fingerprint image correction network are further pruned to obtain a fingerprint image correction model that only includes the coordinate processing branch.

[0056] When formally performing forward inference using the pre-trained and pruned fingerprint image correction model, the original image of the uncorrected non-contact fingerprint is input into the fingerprint image correction model, and the original image is processed by the coordinate processing branch of the fingerprint image correction model to obtain and output the first pixel coordinate remapped coordinates.

[0057] Step 103: Based on the first pixel remapped coordinates, perform remapping processing on the original image to obtain the target image of the corrected non-contact fingerprint.

[0058] The target image is a fingerprint image that can be used for non-contact fingerprint recognition after correcting the original image of the non-contact fingerprint that has undergone perspective distortion due to the curvature of the finger surface during shooting.

[0059] Specifically, after obtaining the first pixel remapping coordinates required to correct the perspective distortion of the original non-contact fingerprint image output by the fingerprint image correction model, the scale of the remapping coordinates is scaled to the same scale as the original image using the bilinear interpolation strategy, and the original image is remapped using the coordinate remapping method to obtain the target image of the corrected non-contact fingerprint.

[0060] The image correction method for non-contact fingerprints provided by the present invention constructs a fingerprint image correction network with an image decoder branch and an image generation module during model pre-training, and uses training samples including non-contact fingerprint image samples, their corresponding remapping coordinate labels, and corrected image labels to train the fingerprint image correction network, so as to utilize the auxiliary supervised training results of the image decoder branch and the image generation module to enable the network structure before the image decoder branch and the image generation module in the fingerprint image correction network to extract better image features. After the pre-training is completed, the image decoder branch and the image generation module are cropped to obtain a fingerprint image correction model for inference, which can better correct the perspective distortion of non-contact fingerprint images on the basis of reducing the hardware requirements, acquisition difficulty, and acquisition cost of the fingerprint original image acquisition device. At the same time, it does not require depth estimation and 3D reconstruction, greatly reducing the requirements for hardware computing resources, having stronger applicability and a wider application range, and being more conducive to improving the recognition accuracy of non-contact fingerprints.

[0061] Based on the above embodiments, as an optional embodiment, the fingerprint image correction network includes a feature extraction module, an encoder module, a decoder module, a coordinate remapping module, and the image generation module; the decoder module includes a coordinate decoder branch and the image decoder branch; The feature extraction module is used to determine fingerprint extraction features based on the non-contact fingerprint image samples; The encoder module is used to determine multi-scale encoder features based on the fingerprint extraction features; The coordinate decoder branch is used to determine coordinate remapping features based on the multi-scale encoder features; The image decoder branch is used to determine multi-scale image features based on the multi-scale encoder features; The coordinate remapping module is used to determine second pixel remapping coordinates based on the coordinate remapping features; The image generation module is used to determine fingerprint correction images based on the multi-scale image features and the fingerprint extraction features.

[0062] The second pixel remapping coordinates are the coordinates of the remapping points obtained after processing the non-contact fingerprint image samples by the feature extraction module, encoder module, coordinate decoder branch, and coordinate remapping module of the fingerprint image correction network during the training stage of the fingerprint image correction network, and are used for remapping the non-contact fingerprint image samples.

[0063] Specifically, Figure 2 is a schematic structural diagram of the fingerprint image correction network provided by the present invention. As Figure 2 shown, the fingerprint image correction network is constructed based on the combination of CNN and Transformer, and includes a feature extraction module, an encoder module, a decoder module, a coordinate remapping module, and an image generation module. The decoder module includes a coordinate decoder branch and an image decoder branch. Among them, the feature extraction module is connected to the encoder module, the coordinate decoder branch in the decoder module is respectively connected to the encoder module and the coordinate remapping module, and the image decoder branch in the decoder module is respectively connected to the encoder module and the image generation module. The coordinate decoder branch and the coordinate remapping module can be collectively referred to as the coordinate processing branch, and the image decoder branch and the image generation module can be collectively referred to as the image processing branch.

[0064] It can be seen that the decoder module in the fingerprint image correction network adopts a dual-branch structure: the coordinate decoder branch is used for generating the pixel remapping coordinates required for subsequent perspective deformation correction of the fingerprint image. During the inference stage, the coordinate decoder branch can be used for generating the first pixel remapping coordinates subsequently; the image decoder branch is used for assisting in supervised training to enable the network structures of the feature extraction module and the encoder module before the image decoder branch to extract better image features.

[0065] Furthermore, the feature extraction module can extract the low-level features of the input fingerprint image. During the training of the fingerprint image correction network, after inputting the non-contact fingerprint image samples in the training samples into the feature extraction module, the feature extraction module processes and obtains and outputs fingerprint extraction features. The encoder module connected to the feature extraction module processes the fingerprint extraction features and obtains and outputs multi-scale encoder features.

[0066] Optionally, the feature extraction module is built based on CNN network models such as Residual Network (ResNet), EfficientNet, and MobileNet.

[0067] The coordinate decoder branch of the decoder module processes the multi-scale encoder features to obtain and output coordinate remapping features, and then the coordinate remapping module connected to the coordinate decoder branch processes the coordinate remapping features to obtain and output the second pixel remapping coordinates. According to the error between the second pixel remapping coordinates and the remapping coordinate labels corresponding to the non-contact fingerprint image samples, the network model parameters of the feature extraction module, the encoder module, the coordinate decoder branch, and the coordinate remapping module can be optimized.

[0068] The image decoder branch of the decoder module processes the multi-scale encoder features to obtain and output multi-scale image features, and then the image generation module connected to the image decoder branch processes the multi-scale image features and the fingerprint extraction features to obtain and output the fingerprint corrected image. According to the error between the fingerprint corrected image and the corrected image labels corresponding to the non-contact fingerprint image samples, the network model parameters of the feature extraction module, the encoder module, the image decoder branch, and the image generation module can be optimized.

[0069] Figure 3 is a schematic structural diagram of the fingerprint image correction model provided by the present invention. As Figure 3 shown, after completing the training of the fingerprint image correction network, the image decoder and the image generation module of the decoder module in the fingerprint image correction network are cropped to obtain a fingerprint image correction model for inference composed of a feature extraction module, an encoder module, a coordinate decoder branch, and a coordinate remapping module. The parameters of the network structures of the feature extraction module and the encoder module are jointly optimized and determined by the training results of the coordinate processing branch and the image processing branch, and can extract better fingerprint image features.

[0070] During the inference of the fingerprint image correction model, the original image of the uncorrected non-contact fingerprint is output to the feature extraction module, and after being processed by the feature extraction module, the encoder module, the coordinate decoder branch, and the coordinate remapping module in sequence, the first pixel remapping coordinates output by the coordinate remapping module are obtained, so as to realize remapping processing of the original image based on the first pixel remapping coordinates to obtain the corrected target image.

[0071] The non-contact fingerprint image correction method provided by the present invention sets up a dual-processing branch in the fingerprint image correction network, that is, a coordinate processing branch including a coordinate decoder branch and a coordinate remapping module and an image processing branch including an image decoder branch and an image generation module are set up simultaneously. During training, the output results of both the image processing branch and the coordinate processing results can be used to optimize the network structure parameters of the feature extraction module and the encoder module for extracting fingerprint image features, so that the fingerprint image correction model after cropping the image processing branch can have better fingerprint image feature extraction ability during inference, thereby improving the correction effect of non-contact fingerprint images; moreover, the image generation module will simultaneously determine the output fingerprint correction image based on the multi-scale image features output by the image decoder branch of the decoder module and the fingerprint extraction features output by the feature extraction module, which can improve the model performance during the network optimization training process.

[0072] Based on the above embodiments, as an optional embodiment, the Transformer structures in the encoder module and the decoder module are constructed based on the multi-head attention mechanism (Multi-Head Attention Mechanism).

[0073] The non-contact fingerprint image correction method provided by the present invention constructs the Transformer structures in the encoder module and the decoder module by adopting the multi-head attention mechanism, which is more conducive to extracting the corresponding relationships between pixels in non-contact fingerprint images and is more conducive to the restoration and correction of fingerprint image perspective distortion.

[0074] Based on the above embodiments, as an optional embodiment, the encoder module includes a first-layer encoder, a first first-convolution layer (i.e., the first CNN layer), a second-layer encoder, a second first-convolution layer, and a third-layer encoder connected in sequence; The first-layer encoder is used to determine the first-scale encoder features based on the fingerprint extraction features; The first first-convolution layer is used to downsample the output features of the first-layer encoder; The second-layer encoder is used to determine the second-scale encoder features based on the output features of the first first-convolution layer; The second first-convolution layer is used to downsample the output features of the second-layer encoder; The third-layer encoder is used to determine the third-scale encoder features based on the output features of the first first-convolution layer.

[0075] The first CNN layer refers to the CNN network structure set in the encoder module.

[0076] Specifically,Figure 4 It is a schematic structural diagram of the encoder module provided by the present invention. As Figure 4 shown, the encoder module is jointly composed of a Transformer structure and a CNN structure based on the multi-head self-attention mechanism, and includes three layers of encoders and two layers of first CNN layers, specifically including a first layer of encoder, a first layer of first CNN layer, a second layer of encoder, a second layer of first CNN layer, and a third layer of encoder connected in sequence. Among them, each layer of encoder includes a number of standard Transformer encoding layers.

[0077] When the encoder module processes the fingerprint extraction features output by the feature extraction module, the fingerprint extraction features are combined with the position encoding. The first layer of encoder determines and outputs the first-scale encoder features based on the fingerprint extraction features. The first layer of first CNN layer downsamples the output features of the first layer of encoder. The second layer of encoder further determines and outputs the second-scale encoder features based on the downsampled output features output by the first layer of first CNN layer. The second layer of first CNN layer downsamples the output features of the second layer of encoder. The third layer of encoder further determines and outputs the third-scale encoder features based on the downsampled output features output by the second layer of first CNN layer.

[0078] Thus, the output of the encoder module also includes multi-scale encoder features composed of the first-scale encoder features, the second-scale encoder features, and the third-scale encoder features.

[0079] Among them, the first-scale encoder features, the second-scale encoder features, and the third-scale encoder features output by the three layers of encoders of the encoder module are the inputs of the corresponding decoders in the image decoder branch of the decoder module to achieve multi-scale feature fusion in the decoder module, and the feature scales of the first-scale encoder features, the second-scale encoder features, and the third-scale encoder features are different.

[0080] Optionally, the scale of the first-scale encoder features is greater than the scale of the second-scale encoder features, and the scale of the second-scale encoder features is greater than the scale of the third-scale encoder features.

[0081] Optionally, when the fingerprint extraction features output by the feature extraction module are multi-scale fingerprint features, the input of the first layer of encoder is the low-scale fingerprint features in the multi-scale fingerprint features, and the first layer of encoder is used to determine the first-scale encoder features based on the low-scale fingerprint features; correspondingly, the image generation module is used to determine the fingerprint correction image based on the multi-scale image features and the fingerprint features other than the low-scale fingerprint features in the multi-scale fingerprint features.

[0082] For example, in the case where the multi-scale fingerprint features output by the feature extraction module include high-scale fingerprint features, medium-scale fingerprint features, and low-scale fingerprint features with gradually decreasing scales, the low-scale fingerprint features will be input into the first layer encoder in the encoder module, and the high-scale fingerprint features and medium-scale fingerprint features will be input into the image generation module.

[0083] The non-contact fingerprint image correction method provided by the present invention composes the encoder module by adopting a Transformer structure and a CNN structure together. Based on the characteristic that the Transformer structure is more conducive to extracting the corresponding relationship between each pixel in the fingerprint image features, the fingerprint image correction network is more conducive to the restoration and correction of the perspective distortion of the fingerprint image; based on the CNN layer, downsampling operation of the feature map output by the encoder can be realized to construct the extraction and fusion of encoder features at different scales; the encoder module is composed of three layers of Transformer-based encoders and two layers of CNN layers, so that the encoder module can not only obtain image features at high resolution, but also obtain image features with high-level semantic information at low resolution, which can optimize the training of the fingerprint image correction network and obtain a fingerprint image correction model with better performance parameters, and is beneficial to the restoration and correction of the perspective distortion of the non-contact fingerprint image.

[0084] Based on the above embodiments, as an optional embodiment, the coordinate decoder branch includes a first-layer coordinate decoder, a first-layer second convolutional layer (i.e., the second CNN layer), a second-layer coordinate decoder, a second-layer second CNN layer, and a third-layer coordinate decoder connected in sequence; The input of the first-layer coordinate decoder is determined based on the first-scale encoder features; The input of the second-layer coordinate decoder is determined based on the second-scale encoder features and the output features of the first-layer second CNN layer; The input of the third-layer coordinate decoder is determined based on the third-scale encoder features and the output features of the second-layer second CNN layer.

[0085] Specifically, Figure 5 is a schematic structural diagram of the decoder module provided by the present invention. As Figure 5 shown, the decoder module is composed of a Transformer structure based on the multi-head self-attention mechanism and a CNN structure together, and is composed of two branches. One branch is the coordinate decoder branch, which is used for the generation of subsequent pixel remapping coordinates; the other branch is the image decoder branch, which is used for the auxiliary supervised training of the fingerprint image correction network during the training process, and better optimizes the networks of the encoder module and the feature extraction module before the decoder module to extract better image features.

[0086] The coordinate decoder branch includes a three-layer coordinate decoder and two-layer second CNN layer, specifically including a first-layer coordinate decoder, a first-layer second CNN layer, a second-layer coordinate decoder, a second-layer second CNN layer, and a third-layer coordinate decoder connected in sequence. When the coordinate decoder branch processes the output of the encoder module, the learned embedding is first combined with the positional encoding. The first-layer coordinate decoder processes the first-scale encoder features output by the first-layer encoder and the learned embedding after combining with the positional encoding to obtain output features. The first-layer second CNN layer upsamples the output features of the first-layer coordinate decoder. The second-layer coordinate decoder processes the second-scale encoder features output by the second-layer encoder and the output features of the first-layer second CNN layer to obtain output features. The second-layer second CNN layer upsamples the output features of the second-layer coordinate decoder. The third-layer coordinate decoder processes the third-scale encoder features output by the third-layer encoder and the output features of the second-layer second CNN layer to obtain the coordinate remapping features finally output by the coordinate decoder branch.

[0087] The image correction method for non-contact fingerprints provided by the present invention can perform image processing by correspondingly combining multi-scale encoder features output by the encoder module by setting the coordinate decoder branch in the decoder module to a structure of a three-layer coordinate decoder and two convolutional layers corresponding to the encoder module. This is beneficial to optimizing the training of the fingerprint image correction network through the multi-scale feature fusion mechanism and obtaining a fingerprint image correction model with better performance parameters, and further beneficial to the restoration and correction of the perspective deformation of non-contact fingerprint images.

[0088] Based on the above embodiments, as an alternative embodiment, the coordinate remapping module includes a remapped coordinate initialization layer, a third convolutional layer, a remapped coordinate fusion layer, and a remapped coordinate upsampling layer; The remapped coordinate initialization layer is used to fixedly initialize the original value of the coordinate remapping features; The third convolutional layer is used to determine the initially fitted initial remapped coordinates based on the coordinate remapping features; The remapped coordinate fusion layer is used to fuse the initial remapped coordinates and the original value to obtain the fused remapped coordinates; The remapped coordinate upsampling layer is used to upsample the fused remapped coordinates in combination with the coordinate remapping features to determine the second pixel remapped coordinates.

[0089] Specifically, Figure 6 is a schematic structural diagram of the coordinate remapping module provided by the present invention, as shown in Figure 6As shown in the figure, the coordinate remapping module includes a remapped coordinate initialization layer, a third convolutional layer, a remapped coordinate fusion layer, and a remapped coordinate upsampling layer. Among them, the remapped coordinate initial layer is respectively connected to the third convolutional layer and the remapped coordinate fusion layer, the third convolutional layer is connected to the remapped coordinate fusion layer, and the remapped coordinate fusion layer is connected to the remapped coordinate upsampling layer.

[0090] The coordinate remapping module is used to generate the pixel remapping coordinates required for the perspective distortion correction of the fingerprint image, that is, to obtain the remapping of the coordinates of the uncorrected fingerprint image corresponding to the corresponding coordinate positions of the corrected fingerprint image.

[0091] Before the third convolutional layer of the coordinate remapping module performs a preliminary fit on the coordinate remapping features input to the coordinate remapping module, the remapped coordinate initialization layer first performs a fixed initialization of the original values of the coordinate remapping features, and then the third convolutional layer performs a preliminary fit on the coordinate remapping features to obtain the initially remapped coordinates of the preliminary fit. The remapped coordinate fusion layer fuses the initially remapped coordinates of the preliminary fit and the original values of the coordinate remapping features after fixed initialization to obtain and output the downsampled and small-scale fused remapped coordinates.

[0092] Furthermore, the remapped coordinate upsampling layer is used to upsample the fused remapped coordinates. Since during the upsampling process, the upsampling of each coordinate value of the fused remapped coordinates needs to focus on the coordinate values in its adjacent region, the coordinate remapping features output by the coordinate decoder branch are used as the output of the remapped coordinate upsampling layer at the same time. The fused remapped coordinates are fused and upsampled with the surrounding eigenvalue in combination with the coordinate remapping features to obtain the second pixel remapping coordinates with the same size as the non-contact fingerprint image sample.

[0093] Finally, according to the error between the second pixel remapping coordinates and the remapping coordinate labels corresponding to the non-contact fingerprint image sample, the network model parameters of the feature extraction module, the encoder module, the coordinate decoder branch, and the coordinate remapping module can be optimized.

[0094] The non-contact fingerprint image correction method provided by the present invention performs fixed initialization of the original values of the coordinate remapping features by adopting a fixed initialization method instead of the randomly initialized method commonly used in neural networks. It fully considers that the coordinate mapping changes caused by mirror distortion are coordinate transformations around the corresponding positions, and there will be no sudden changes or drastic changes in the mapping value relationship. By initializing with the default non-distorted situation, it can ensure that each remapped coordinate conforms to the change law, which is more conducive to the training of the fingerprint image correction network. At the same time, by upsampling the fused remapped coordinates in combination with the coordinate remapping features, it can fully consider the relationship between the coordinate values in the fused remapped coordinates and the coordinate values in the adjacent regions, which is more conducive to the training of the fingerprint image correction network and can obtain a fingerprint image correction model with better performance parameters, thereby facilitating the restoration and correction of the perspective distortion of non-contact fingerprint images.

[0095] Based on the above embodiments, as an alternative embodiment, the image decoder branch includes a first-layer image decoder, a first-layer fourth convolutional layer (i.e., the fourth CNN layer), a second-layer image decoder, a second-layer fourth CNN layer, and a third-layer image decoder connected in sequence; The input of the first-layer image decoder is determined based on the first-scale encoder features, and the output of the first-layer image decoder is the first-scale image features; The input of the second-layer image decoder is determined based on the second-scale encoder features and the output features of the first-layer fourth CNN layer, and the output of the second-layer image decoder is the second-scale image features; The input of the third-layer image decoder is determined based on the third-scale encoder features and the output features of the second-layer fourth CNN layer, and the output of the third-layer image decoder is the third-scale image features.

[0096] Specifically, as shown in Figure 5 the image decoder branch includes three layers of image decoders and two layers of fourth CNN layers, specifically including a first-layer image decoder, a first-layer fourth CNN layer, a second-layer image decoder, a second-layer second CNN layer, and a third-layer image decoder connected in sequence. When the image decoder branch processes the output of the encoder module, the learned embedding is first combined with the positional encoding. It can be understood that the positional encoding used by the coordinate decoder branch and the image decoder branch in the same decoder module to combine the learned embedding is the same.

[0097] The first-layer image coordinate decoder of the image decoder branch processes the first-scale encoder features output by the first-layer encoder and the learned embedding combined with positional encoding, and obtains and outputs the first-scale image features; the fourth CNN layer of the first layer upsamples the output features of the first-layer image coordinate decoder; the second-layer image coordinate decoder processes the second-scale encoder features output by the second-layer encoder and the output features of the fourth CNN layer of the first layer, and obtains and outputs the second-scale image features; the fourth CNN layer of the second layer upsamples the output features of the second-layer image coordinate decoder; the third-layer image coordinate decoder processes the third-scale encoder features output by the third-layer encoder and the output features of the fourth CNN layer of the second layer, and obtains and outputs the third-scale image features.

[0098] Thus, the output of the image decoder branch is also the multi-scale image features composed of the first-scale image features, the second-scale image features, and the third-scale image features.

[0099] The image correction method for non-contact fingerprints provided by the present invention can perform image processing by correspondingly combining the multi-scale encoder features output by the encoder module by setting the image decoder branch in the decoder module to a structure of three-layer image decoders corresponding to the encoder module and two convolutional layers for upsampling, which is beneficial to optimizing the training of the fingerprint image correction network through the multi-scale feature fusion mechanism and obtaining a fingerprint image correction model with better performance parameters, and further beneficial to the restoration and correction of the perspective deformation of the non-contact fingerprint image.

[0100] Based on the above embodiments, as an optional embodiment, the image generation module includes a first-layer upsampling convolutional layer, a second-layer upsampling convolutional layer, a third-layer upsampling convolutional layer, a fourth-layer upsampling convolutional layer, and a fifth-layer upsampling convolutional layer connected in sequence; The input of the first-layer upsampling convolutional layer is determined based on the first-scale image features; The input of the second-layer upsampling convolutional layer is determined based on the output features of the first-layer upsampling convolutional layer and the second-scale image features; The input of the third-layer upsampling convolutional layer is determined based on the output features of the second-layer upsampling convolutional layer and the third-scale image features; The input of the fourth-layer upsampling convolutional layer is determined based on the output features of the third-layer upsampling convolutional layer and the fingerprint extraction features; The input of the fifth-layer upsampling convolutional layer is determined based on the output features of the fourth-layer upsampling convolutional layer and the fingerprint extraction features.

[0101] Specifically, Figure 7 is a schematic structural diagram of the image generation module provided by the present invention, as Figure 7As shown, the image generation module is used to output a fingerprint correction image after perspective distortion correction of the uncorrected non-contact fingerprint image sample, and the fingerprint correction image is used for the auxiliary supervised training of the fingerprint image correction network. The image generation module adopts a multi-scale structure similar to the UNet network, specifically including five upsampling convolutional layers connected in sequence, namely the first upsampling convolutional layer, the second upsampling convolutional layer, the third upsampling convolutional layer, the fourth upsampling convolutional layer, and the fifth upsampling convolutional layer.

[0102] The image generation module obtains the fingerprint correction image based on the processing of the multi-scale image features output by the image decoder branch of the decoder module and the fingerprint extraction features output by the feature extraction module. Specifically, the input of the first upsampling convolutional layer is the first-scale image features output by the first image decoder in the image decoder branch, and the output features are obtained after processing; the input of the second upsampling convolutional layer is the second-scale image features output by the second image decoder in the image decoder branch and the output features of the first upsampling convolutional layer, and the output features are obtained after processing; the input of the third upsampling convolutional layer is the third-scale image features output by the third image decoder in the image decoder branch and the output features of the second upsampling convolutional layer, and the output features are obtained after processing; the input of the fourth upsampling convolutional layer is the fingerprint extraction features output by the feature extraction module in the image decoder branch and the output features of the third upsampling convolutional layer, and the output features are obtained after processing; the input of the fifth upsampling convolutional layer is the fingerprint extraction features output by the feature extraction module and the output features of the fourth upsampling convolutional layer, and the fingerprint correction image finally output by the image generation module is obtained after processing.

[0103] Finally, according to the error between the fingerprint correction image and the correction image label corresponding to the non-contact fingerprint image sample, the network model parameters of the feature extraction module, the encoder module, the image decoder branch, and the image generation module can be optimized. After the fingerprint image correction network training is completed, the image decoder branch and the image generation module will be pruned to obtain a fingerprint image correction model composed of the feature extraction module, the encoder module, the coordinate decoder branch, and the coordinate remapping module. And under the auxiliary supervised training of the image decoder branch and the image generation module, the feature extraction module and the encoder module have better image feature extraction capabilities.

[0104] The image correction method for non-contact fingerprints provided by the present invention processes the multi-scale image features output by the image decoder branch and the fingerprint extraction features output by the feature extraction module by using the image generation module to obtain the fingerprint correction image of the non-contact fingerprint image sample, and then enables the feature extraction module and the decoder module to obtain better training parameters under the action of auxiliary supervised training, which is beneficial to the restoration and correction of the perspective distortion of the non-contact fingerprint image.

[0105] Based on the above embodiment, as an optional embodiment, the multiple training samples are obtained based on the following method: Obtaining a first number of the training samples based on a manual remapping coordinate annotation method; Based on the automatic generation of remapped coordinates, obtaining a second number of the training samples; The first number is smaller than the second number.

[0106] The manual remapping coordinate annotation method refers to first obtaining the non-contact fingerprint image samples in the training samples, and then obtaining the remapping coordinate labels and corrected image labels corresponding to the non-contact fingerprint image samples by manual annotation to obtain the training samples; the automatic remapping coordinate generation method refers to first obtaining the corrected image labels in the training samples, and then using a pre-programmed program to automatically generate remapping coordinate labels and automatically perform reverse remapping processing to obtain non-contact fingerprint image samples to obtain the training samples.

[0107] Specifically, when obtaining training samples for training the fingerprint image correction network, if only manual annotation is used for data annotation, a large amount of labor cost overhead will be incurred. Therefore, it is necessary to combine manual annotation and automatic generation methods to obtain a small number of first number of training samples obtained by manual annotation based on remapping coordinates, and a large number of second number of training samples obtained by automatically generating samples based on remapping coordinates, to obtain the total training samples for training the fingerprint image correction network.

[0108] The non-contact fingerprint image correction method provided by the present invention simultaneously obtains a small amount of training samples by manually annotating data and obtains a large amount of training samples by automatically generating label data. While obtaining a large amount of training samples by automatically generating them to reduce the cost of obtaining training samples as much as possible, a small amount of real training samples is obtained by manually annotating them to ensure that the weight parameters finally learned by the fingerprint image correction network training are more accurate and will not have deviations.

[0109] Based on the above embodiment, as an optional embodiment, the step of obtaining the first number of training samples based on the manual annotation method of remapping coordinates includes: Acquire the first number of uncorrected contactless fingerprint images as the contactless fingerprint image samples; Manually marking the remapped coordinate points of the uncorrected contactless fingerprint image to obtain the remapped coordinate labels corresponding to the contactless fingerprint image sample; Remapping the uncorrected contactless fingerprint image based on the remapping coordinate label to obtain the corrected image label corresponding to the contactless fingerprint image sample; Determine the first quantity of the training samples based on the non-contact fingerprint image samples, the remapped coordinate labels, and the corrected image labels.

[0110] Specifically, when obtaining the first quantity of training samples by means of manual annotation based on remapped coordinates, a non-contact fingerprint image of a finger is normally captured using an imaging device such as a mobile phone, and the first quantity of uncorrected non-contact fingerprint images are obtained as the first quantity of non-contact fingerprint image samples.

[0111] For any non-contact fingerprint image sample, manually annotate the remapped coordinate points of the uncorrected non-contact fingerprint image sample to obtain the remapped coordinate label corresponding to this non-contact fingerprint image sample; perform remapping processing on this non-contact fingerprint image sample using the manually annotated remapped coordinate label to obtain the corrected image label of this non-contact fingerprint image sample; this non-contact fingerprint image sample and its corresponding remapped coordinate label and corrected image label are combined together to obtain a training sample.

[0112] Repeat the steps of manually annotating to obtain the remapped coordinate label and remapping processing to obtain the corrected image label, and the first quantity of manually annotated training samples can be obtained. Since manual annotation is required and the labor cost is relatively high, only a small amount of the first quantity of training samples are prepared using this method.

[0113] Based on the above embodiments, as an optional embodiment, the method for obtaining the second quantity of the training samples by means of automatic generation based on remapped coordinates includes: Obtain the third quantity of approximately corrected contact fingerprint images as the corrected image labels; Automatically generate the remapped coordinate points of the approximately corrected contact fingerprint images according to a preset fingerprint perspective deformation rule to obtain the second quantity of the remapped coordinate labels; Perform inverse remapping processing on the approximately corrected contact fingerprint images based on the remapped coordinate labels to obtain the second quantity of the non-contact fingerprint image samples; Determine the second quantity of the training samples based on the second quantity of the non-contact fingerprint image samples, the second quantity of the remapped coordinate labels, and the third quantity of the corrected image labels; The third quantity is less than the second quantity.

[0114] Specifically, collect the third quantity of approximately corrected contact fingerprint images through imaging devices such as mobile phones as the corrected image labels in the training samples; then automatically generate a number of remapping coordinate points of the approximately corrected contact fingerprint images according to the preset fingerprint perspective deformation rules through a pre-programmed program. Since different remapping coordinate points can be generated for the same approximately corrected contact fingerprint image according to different preset fingerprint perspective deformation rules, a second quantity of remapping coordinate labels greater than the third quantity can be obtained. For example, for 1 approximately corrected contact fingerprint image, 10 different groups of remapping coordinate points can be generated according to different preset fingerprint perspective deformation rules.

[0115] Repeatedly use the remapping coordinate labels to perform inverse remapping processing on the corresponding approximately corrected contact fingerprint images, that is, perform an operation opposite to the remapping operation, that is, generate the uncorrected fingerprint image corresponding to the approximately corrected contact fingerprint image and the remapping coordinate label as the non-contact fingerprint image sample. A second quantity of non-contact fingerprint image samples equal to the number of remapping coordinate labels can be obtained. Combine the second quantity of non-contact fingerprint image samples, the second quantity of remapping coordinate labels, and the third quantity of corrected image labels together to obtain the second quantity of training samples, where there are several cases where the corrected image labels of the training samples are the same, and the non-contact fingerprint image samples and remapping coordinate labels are different.

[0116] Figure 8 It is an acquisition example diagram of the approximately corrected contact fingerprint image provided by the present invention. As Figure 8 shown, press the finger forcefully on a rigid transparent material plate such as a high-definition transparent glass to make the finger pulp press into a plane, and use an imaging device such as a mobile phone to take a picture of the finger pressed into a plane through the rigid transparent material plate. At the same time, pay attention to the absence of light source interference such as reflection of the glass during the shooting process, and obtain the initial shooting image of the contact fingerprint. Cut out the finger pulp part of the initial shooting image pressed into a plane as the approximately corrected contact fingerprint image.

[0117] The non-contact fingerprint image correction method provided by the present invention can batch generate a large number of paired training samples by collecting a small number of contact fingerprint images as the corrected image labels after approximate correction of the non-contact fingerprint images, then automatically generating a larger number of remapping coordinate labels according to the preset fingerprint perspective deformation rules, and performing inverse remapping operations on the corrected image labels according to the remapping coordinate labels to obtain the corresponding non-contact fingerprint image samples. In particular, each approximately corrected corrected image label can batch generate different remapping coordinate labels and non-contact fingerprint image samples, which solves the difficulty of preparing the training sample data annotation, greatly reduces the labor cost of data annotation, and also performs data augmentation on the training samples.

[0118] To better illustrate the non-contact fingerprint image correction method provided by the present invention, an embodiment is provided below to illustrate the training and inference processes in the non-contact fingerprint image correction method.

[0119] First, obtain the training samples required for training the fingerprint image correction network. Using the aforementioned manual annotation method of remapped coordinates and the automatic generation method of remapped coordinates, obtain multiple training samples. When manually annotating the remapped coordinate points of the uncorrected non-contact fingerprint image in the manual annotation method of remapped coordinates, after performing grid annotation with 16×16 key points, convert them into 288×288 key points using the bilinear interpolation method, and ensure that the offset positions of each key point are mutually restricted, conforming to the perspective distortion of the real non-contact fingerprint image.

[0120] After obtaining multiple training samples, build a fingerprint image correction network according to the network structures of the aforementioned feature extraction module, encoder module, decoder module (including coordinate decoder branch and image decoder branch), coordinate remapping module, and image generation module.

[0121] Among them, the feature extraction module adopts common network model structures such as ResNet, EfficientNet, MobileNet, etc. to extract the image features of the original image of the uncorrected non-contact fingerprint or the non-contact fingerprint image sample. The image scale input to the feature extraction module is 288×288 resolution, and the fingerprint extraction features output by the feature extraction module are multi-scale fingerprint features. Specifically, the features with a width and height scale of 1 / 2, 1 / 4, and 1 / 8 of the input width and height scales are output as the output of the feature extraction module, that is, the output scales are the high-scale fingerprint features of 144×144, the medium-scale fingerprint features of 72×72, and the low-scale fingerprint features of 36×36 respectively. Among them, the high-scale fingerprint features of 144×144 and the medium-scale fingerprint features of 72×72 will be used as the inputs of the fourth upsampling convolutional layer and the fifth upsampling convolutional layer of the image generation module respectively, and the low-scale fingerprint features of 36×36 will be used as the input of the encoder module.

[0122] In the Transformer structure of each layer of the encoder in the encoder module, the multi-head attention mechanism is adopted. The number of hidden layers is 256. Each layer of the encoder uses 2 self-attention layers, the number of attention heads is 8, the output number of the feed-forward layer is 2048, and Dropout is 0.1. Each first convolutional layer in the encoder module serves the purpose of downsampling, which is used to reduce the scale of the feature map input to the first convolutional layer to 1 / 2 of the input. It is implemented based on the ordinary Conv2D convolution with a stride of 2 and a convolution kernel size of 3*3. The position encoding is encoded using sine and cosine functions. After the 36*36 low-scale fingerprint features are input into the encoder module, the first layer of the encoder outputs the first-scale encoder features with a scale of 36*36, the second layer of the encoder outputs the second-scale encoder features with a scale of 18*18, and the third layer of the encoder outputs the third-scale encoder features with a scale of 9*9.

[0123] In the Transformer structure of each layer of the image decoder and the coordinate decoder in the decoder module, the multi-head attention mechanism is also adopted. The number of hidden layers is 256. Each layer of the decoder uses 2 self-attention layers, the number of attention heads is 8, the output number of the feed-forward layer is 2048, and Dropout is 0.1. Each second convolutional layer in the decoder module serves the purpose of upsampling, which is used to upsample the scale of the feature map input to the second convolutional layer to 2 times the input. It is implemented based on the transposed convolution with a convolution kernel size of 2*2. The position encoding is encoded using sine and cosine functions. The inputs of the image decoder branch and the coordinate decoder branch in the decoder module are the first learned embedding and the second learned embedding respectively, and the number of embeddings is 81 for both. The multi-scale image features of the image decoder branch are the first-scale image features with a scale of 36*36 output by the first layer of the image decoder, the second-scale image features with a scale of 18*18 output by the second layer of the image decoder, and the third-scale image features with a scale of 9*9 output by the third layer of the image decoder. Correspondingly, they will be used as the inputs of the first upsampling convolutional layer, the second upsampling convolutional layer, and the third upsampling convolutional layer of the image generation module respectively. The feature scale of the coordinate remapping features output by the coordinate decoder branch is 36*36, which will be used as the input of the coordinate remapping module.

[0124] The image generation module adopts a multi-scale structure similar to the UNet network. Its inputs are multi-scale first-scale image features, second-scale image features, third-scale image features, high-scale fingerprint features, and medium-scale fingerprint features, with scales of 9*9, 18*18, 36*36, 72*72, and 144*144 in sequence. Each upsampling convolutional layer in the image generation module mainly completes feature upsampling at different scales and the fusion of corresponding-scale features. Each upsampling convolutional layer upsamples the input features to twice the size of the input, implemented based on a deconvolution with a 2*2 convolutional kernel size. Finally, the output scale of the image generation module is the same as the image scale of the input feature extraction module, that is, a fingerprint correction image with a scale of 288*288.

[0125] The remapping coordinate initialization layer in the coordinate remapping module initializes the size of the coordinate remapping features output by the coordinate decoder branch to a scale of 2*36*36. The remapping coordinate upsampling layer upsamples the coordinates of the 2*36*36 scale to a scale of 2*288*288 after upsampling processing.

[0126] Furthermore, the CrossEntropyLoss loss function is selected, and the AdamW optimizer is used for iterative optimization. The initial learning rate is selected as 0.0001, and the cosine annealing learning rate adjustment strategy is adopted. The fingerprint image correction network is gradually iteratively trained using training samples. During the training process, the feature inputs of the encoder module of the fingerprint image correction network are randomly covered with masks, and the mask ratio coefficient is 0.1.

[0127] After the fingerprint image correction network is trained, the image decoder branch and the image generation module in the fingerprint image correction network are cropped to obtain a fingerprint image correction model for inference.

[0128] In the model inference stage for image correction of non-contact fingerprints, the original image of the uncorrected non-contact fingerprint obtained is input into the fingerprint image correction model for inference calculation to obtain the first pixel remapping coordinates required to correct the original image. Further, the bilinear interpolation strategy is used to scale the size of the remapping coordinates to the same scale as the original image, and the original image is remapped based on the first pixel remapping coordinates to obtain the target image of the corrected non-contact fingerprint, which is beneficial to improving the accuracy of non-contact fingerprint recognition based on the corrected target image.

[0129] Figure 9 It is a schematic structural diagram of the image correction device for non-contact fingerprints provided by the present invention, as Figure 9 shown, the image correction device for non-contact fingerprints includes, but is not limited to, an uncorrected image acquisition module 901, a remapping coordinate acquisition module 902, and a fingerprint remapping processing module 903.

[0130] An uncorrected image acquisition module 901 for acquiring an original image of an uncorrected non-contact fingerprint.

[0131] A remapping coordinate acquisition module 902 for inputting the original image into a fingerprint image correction model to obtain first pixel remapping coordinates output by the fingerprint image correction model.

[0132] A fingerprint remapping processing module 903 for performing remapping processing on the original image based on the first pixel remapping coordinates to obtain a target image of the corrected non-contact fingerprint.

[0133] The fingerprint image correction model is obtained by cropping an image decoder branch and an image generation module in a fingerprint image correction network; the fingerprint image correction network is pre-trained based on a plurality of training samples; each training sample includes a non-contact fingerprint image sample and its corresponding remapping coordinate label and corrected image label.

[0134] It should be noted that the image correction device for non-contact fingerprints provided by the present invention can execute the image correction method for non-contact fingerprints described in any of the above embodiments during specific operation, which will not be elaborated in this embodiment.

[0135] The image correction device for non-contact fingerprints provided by the present invention constructs a fingerprint image correction network with an image decoder branch and an image generation module during model pre-training, and trains the fingerprint image correction network using training samples including non-contact fingerprint image samples and their corresponding remapping coordinate labels and corrected image labels, so as to utilize the auxiliary supervision training results of the image decoder branch and the image generation module to enable the network structure before the image decoder branch and the image generation module in the fingerprint image correction network to extract better image features. After pre-training is completed, the image decoder branch and the image generation module are cropped to obtain a fingerprint image correction model for inference, which can better correct the perspective distortion of non-contact fingerprint images on the basis of reducing the hardware requirements, acquisition difficulty and acquisition cost of fingerprint original image acquisition devices, and at the same time, there is no need for depth estimation and three-dimensional reconstruction, greatly reducing the requirements for hardware computing resources, having stronger applicability and a wider application range, and being more conducive to improving the recognition accuracy of non-contact fingerprints.

[0136] Figure 10 is a schematic structural diagram of an electronic device provided by the present invention, as Figure 10As shown in the figure, the electronic device may include: a processor 1010, a communications interface 1020, a memory 1030, and a communication bus 1040. Among them, the processor 1010, the communications interface 1020, and the memory 1030 communicate with each other through the communication bus 1040. The processor 1010 may call the logical instructions in the memory 1030 to execute the non-contact fingerprint image correction method provided in any of the above embodiments. The non-contact fingerprint image correction method includes, but is not limited to, the following steps: obtaining an original image of an uncorrected non-contact fingerprint; inputting the original image into a fingerprint image correction model to obtain first pixel remapping coordinates output by the fingerprint image correction model; performing a remapping process on the original image based on the first pixel remapping coordinates to obtain a target image of the corrected non-contact fingerprint; the fingerprint image correction model is obtained after cropping the image decoder branch and the image generation module in a fingerprint image correction network; the fingerprint image correction network is pre-trained based on a plurality of training samples; each training sample includes a non-contact fingerprint image sample, its corresponding remapping coordinate label, and a corrected image label.

[0137] In addition, when the logical instructions in the above-mentioned memory 1030 are implemented in the form of a software functional unit and sold or used as an independent product, they may be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, may be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The foregoing storage medium includes: various media such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disc that can store program codes.

[0138] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the non-contact fingerprint image correction method provided in any of the above embodiments. The non-contact fingerprint image correction method includes, but is not limited to, the following steps: obtaining an original image of an uncorrected non-contact fingerprint; inputting the original image into a fingerprint image correction model to obtain a first pixel remapping coordinate output by the fingerprint image correction model; performing a remapping process on the original image based on the first pixel remapping coordinate to obtain a target image of the corrected non-contact fingerprint; the fingerprint image correction model is obtained after cropping the image decoder branch and the image generation module in the fingerprint image correction network; the fingerprint image correction network is pre-trained based on a plurality of training samples; each training sample includes a non-contact fingerprint image sample and its corresponding remapping coordinate label and corrected image label.

[0139] In another aspect, the present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the non-contact fingerprint image correction method provided in any of the above embodiments. The non-contact fingerprint image correction method includes, but is not limited to, the following steps: obtaining an original image of an uncorrected non-contact fingerprint; inputting the original image into a fingerprint image correction model to obtain a first pixel remapping coordinate output by the fingerprint image correction model; performing a remapping process on the original image based on the first pixel remapping coordinate to obtain a target image of the corrected non-contact fingerprint; the fingerprint image correction model is obtained after cropping the image decoder branch and the image generation module in the fingerprint image correction network; the fingerprint image correction network is pre-trained based on a plurality of training samples; each training sample includes a non-contact fingerprint image sample and its corresponding remapping coordinate label and corrected image label.

[0140] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated. The components shown as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. A person of ordinary skill in the art can understand and implement it without creative labor.

[0141] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on such an understanding, the essence of the above technical solution, or the part that contributes to the prior art, can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0142] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements for some of the technical features. And these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A non-contact fingerprint image correction method, characterized in that: include: Acquire an uncorrected original image of the contactless fingerprint; Inputting the original image into a fingerprint image correction model to obtain a first pixel remapping coordinate output by the fingerprint image correction model; Remapping the original image based on the first pixel remapping coordinates to obtain a corrected target image of the contactless fingerprint; The fingerprint image correction model is obtained after cutting out the image decoder branch and the image generation module in the fingerprint image correction network; the fingerprint image correction network is obtained by pre-training based on multiple training samples; each training sample includes a non-contact fingerprint image sample and its corresponding remapped coordinate label and a corrected image label.

2. The non-contact fingerprint image correction method according to claim 1, characterized in that: The fingerprint image correction network includes a feature extraction module, an encoder module, a decoder module, a coordinate remapping module and the image generation module; the decoder module includes a coordinate decoder branch and the image decoder branch; The feature extraction module is used to determine fingerprint extraction features based on the non-contact fingerprint image sample; The encoder module is used to determine a multi-scale encoder feature based on the fingerprint extraction feature; The coordinate decoder branch is configured to determine a coordinate remapping feature based on the multi-scale encoder feature; The image decoder branch is configured to determine a multi-scale image feature based on the multi-scale encoder feature; The coordinate remapping module is used to determine a second pixel remapping coordinate based on the coordinate remapping feature; The image generation module is used to determine a fingerprint correction image based on the multi-scale image features and the fingerprint extraction features.

3. The non-contact fingerprint image correction method according to claim 2, characterized in that: The Transformer structure in the encoder module and the decoder module is constructed based on a multi-head attention mechanism.

4. The non-contact fingerprint image correction method according to claim 2, characterized in that: The encoder module includes a first-layer encoder, a first-layer first convolutional layer, a second-layer encoder, a second-layer first convolutional layer, and a third-layer encoder connected in sequence; The first layer encoder is used to determine a first scale encoder feature based on the fingerprint extraction feature; The first convolutional layer of the first layer is used to downsample the output features of the encoder of the first layer; The second layer encoder is used to determine a second scale encoder feature based on the output feature of the first convolutional layer of the first layer; The first convolutional layer of the second layer is used to downsample the output features of the encoder of the second layer; The third-layer encoder is used to determine a third-scale encoder feature based on the output feature of the first convolutional layer of the first layer.

5. The non-contact fingerprint image correction method according to claim 4, characterized in that: The coordinate decoder branch includes a first-layer coordinate decoder, a first-layer second convolutional layer, a second-layer coordinate decoder, a second-layer second convolutional layer and a third-layer coordinate decoder connected in sequence; The input of the first layer coordinate decoder is determined based on the first scale encoder features; The input of the second layer coordinate decoder is determined based on the second scale encoder feature and the output feature of the second convolutional layer of the first layer; The input of the third layer coordinate decoder is determined based on the third scale encoder features and the output features of the second convolutional layer of the second layer.

6. The non-contact fingerprint image correction method according to claim 4 or 5, characterized in that: The coordinate remapping module includes a remapping coordinate initialization layer, a third convolution layer, a remapping coordinate fusion layer and a remapping coordinate upsampling layer; The remapping coordinate initialization layer is used to fix and initialize the original value of the coordinate remapping feature; The third convolutional layer is used to determine initial remapped coordinates of preliminary fitting based on the coordinate remap features; The remapped coordinate fusion layer is used to fuse the initial remapped coordinates and the original values ​​to obtain fused remapped coordinates; The remapped coordinate upsampling layer is used to upsample the fused remapped coordinates in combination with the coordinate remap features to determine the second pixel remapped coordinates.

7. The non-contact fingerprint image correction method according to claim 4, characterized in that: The image decoder branch includes a first layer image decoder, a first layer fourth convolutional layer, a second layer image decoder, a second layer fourth convolutional layer and a third layer image decoder which are connected in sequence; The input of the first layer image decoder is determined based on the first scale encoder feature, and the output of the first layer image decoder is the first scale image feature; The input of the second layer image decoder is determined based on the second scale encoder feature and the output feature of the fourth convolutional layer of the first layer, and the output of the second layer image decoder is the second scale image feature; The input of the third layer image decoder is determined based on the third scale encoder features and the output features of the second fourth convolutional layer, and the output of the third layer image decoder is the third scale image features.

8. The non-contact fingerprint image correction method according to claim 7, characterized in that: The image generation module includes a first upsampling convolution layer, a second upsampling convolution layer, a third upsampling convolution layer, a fourth upsampling convolution layer and a fifth upsampling convolution layer which are connected in sequence; The input of the first upsampling convolution layer is determined based on the first scale image feature; The input of the second upsampling convolution layer is determined based on the output features of the first upsampling convolution layer and the second-scale image features; The input of the third upsampling convolution layer is determined based on the output features of the second upsampling convolution layer and the third scale image features; The input of the fourth upsampling convolution layer is determined based on the output features of the third upsampling convolution layer and the fingerprint extraction features; The input of the fifth upsampling convolution layer is determined based on the output features of the fourth upsampling convolution layer and the fingerprint extraction features.

9. The non-contact fingerprint image correction method according to claim 1, characterized in that: The multiple training samples are obtained based on the following method: Obtaining a first number of the training samples based on a manual remapping coordinate annotation method; Based on the automatic generation of remapped coordinates, obtaining a second number of the training samples; The first number is smaller than the second number.

10. The non-contact fingerprint image correction method according to claim 9, characterized in that: The step of obtaining the first number of training samples based on the manual annotation method of the remapped coordinates includes: Acquire the first number of uncorrected contactless fingerprint images as the contactless fingerprint image samples; Manually marking the remapped coordinate points of the uncorrected contactless fingerprint image to obtain the remapped coordinate labels corresponding to the contactless fingerprint image sample; Remapping the uncorrected contactless fingerprint image based on the remapping coordinate label to obtain the corrected image label corresponding to the contactless fingerprint image sample; The first number of the training samples is determined based on the non-contact fingerprint image samples, the remapped coordinate labels and the rectified image labels.

11. The non-contact fingerprint image correction method according to claim 9, characterized in that: The step of obtaining the second number of training samples based on the automatic generation of remapped coordinates includes: Acquire a third number of approximately corrected contact fingerprint images as the corrected image labels; Automatically generate remapped coordinate points of the approximately corrected contact fingerprint image according to a preset fingerprint perspective deformation rule to obtain the second number of remapped coordinate labels; Performing reverse remapping processing on the approximately corrected contact fingerprint image based on the remapping coordinate label to obtain the second number of the non-contact fingerprint image samples; determining the second number of the training samples based on the second number of the non-contact fingerprint image samples, the second number of the remapped coordinate labels, and the third number of the rectified image labels; The third number is smaller than the second number.

12. A non-contact fingerprint image correction device, characterized in that: include: An uncorrected image acquisition module, used to acquire an uncorrected original image of a contactless fingerprint; A remapping coordinate acquisition module, used for inputting the original image into a fingerprint image correction model to obtain a first pixel remapping coordinate output by the fingerprint image correction model; A fingerprint remapping processing module, configured to perform a remapping process on the original image based on the first pixel remapping coordinates to obtain a corrected target image of the contactless fingerprint; The fingerprint image correction model is obtained after cutting out the image decoder branch and the image generation module in the fingerprint image correction network; the fingerprint image correction network is obtained by pre-training based on multiple training samples; each training sample includes a non-contact fingerprint image sample and its corresponding remapped coordinate label and a corrected image label.

13. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the non-contact fingerprint image correction method according to any one of claims 1 to 11 is implemented.

14. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the non-contact fingerprint image correction method according to any one of claims 1 to 11 is implemented.

15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the non-contact fingerprint image correction method according to any one of claims 1 to 11 is implemented.

Images