Fingerprint image segmentation method, device, non-volatile storage medium and electronic device

Abstract

The present application discloses a fingerprint image segmentation method, apparatus, non-volatile storage medium, and electronic device. The method comprises: determining a fingerprint representation matrix for a fingerprint image to be processed, wherein the fingerprint representation matrix is ​​composed of fingerprint representation vectors determined based on the pixel values ​​of pixels in the fingerprint image to be processed; inputting the fingerprint representation matrix into a fingerprint image segmentation network, and obtaining a confidence map output by the fingerprint image segmentation network, wherein the fingerprint image segmentation network comprises a plurality of sequentially connected attention blocks and a target multi-layer perceptron connected to the last of the plurality of sequentially connected attention blocks; and generating a segmentation mask for the fingerprint image to be processed based on the confidence map. The present application solves the technical problem of poor accuracy and robustness when using threshold segmentation and edge detection to segment fingerprint images on-site due to poor image quality of the images.

Description

Technical Field

[0001] The present application relates to the field of image processing, and more specifically, to a fingerprint image segmentation method, device, non-volatile storage medium, and electronic device. Background Art

[0002] In the prior art, when segmenting fingerprint images collected at the scene, a threshold is typically set to distinguish the fingerprint area from the background area in the image, and then edge detection is used to achieve the segmentation of the fingerprint image. The problem with this approach is that the image quality of the scene image is poor, often suffering from low contrast, high noise, and complex lighting conditions. This leads to limitations in the accuracy and robustness of the fingerprint image segmentation using threshold segmentation and edge detection.

[0003] To address the above-mentioned problems, no effective solutions have been proposed so far. Summary of the Invention

[0004] The embodiments of the present application provide a fingerprint image segmentation method, device, non-volatile storage medium and electronic device to at least solve the technical problem that the accuracy and robustness of segmenting on-site fingerprint images using threshold segmentation and edge detection are poor due to the poor image quality of on-site fingerprint images.

[0005] According to one aspect of an embodiment of the present application, a fingerprint image segmentation method is provided, comprising: determining a fingerprint representation matrix of a fingerprint image to be processed, wherein the fingerprint representation matrix is ​​composed of fingerprint representation vectors determined based on pixel values ​​of pixels in the fingerprint image to be processed; inputting the fingerprint representation matrix into a fingerprint image segmentation network, and obtaining a confidence map output by the fingerprint image segmentation network, wherein the confidence map is used to indicate the probability that each pixel in the fingerprint image to be processed belongs to a fingerprint area, the fingerprint image segmentation network comprising a plurality of sequentially connected attention blocks, and a target multilayer perceptron connected to the last attention block of the plurality of sequentially connected attention blocks; generating a segmentation mask of the fingerprint image to be processed based on the confidence map, wherein the segmentation mask is used to segment the fingerprint image area and the non-fingerprint image area in the fingerprint image to be processed.

[0006] Optionally, the step of determining the fingerprint representation matrix of the fingerprint image to be processed includes: dividing the fingerprint image to be processed into multiple image blocks of the same size, wherein there is no overlapping area between any two image blocks of the multiple image blocks of the same size; determining the fingerprint representation vector corresponding to each image block based on the pixel values ​​of the pixels in each image block, wherein the dimension of the fingerprint representation vector is the same as the number of pixels in the image block; and combining the fingerprint representation vectors corresponding to each image block to obtain the fingerprint representation matrix.

[0007] Optionally, the multiple sequentially connected attention blocks include a first attention block, a second attention block, a third attention block and a fourth attention block connected in sequence, wherein the number of attention heads of the first attention block is 2, representing a dimension of 64, the number of attention heads of the second attention block is 2, representing a dimension of 128, the number of attention heads of the third attention block is 4, representing a dimension of 256, and the number of attention heads of the fourth attention block is 4, representing a dimension of 512.

[0008] Optionally, the first attention block includes a linear embedding layer and two feature extraction layers consisting of a self-attention mechanism module and a multilayer perceptron. The second attention block, the third attention block and the fourth attention block each include a downsampling layer and two feature extraction layers consisting of a self-attention mechanism module and a multilayer perceptron. The fourth attention block is connected to the target multilayer perceptron.

[0009] Optionally, the fingerprint image segmentation network is trained in the following manner: determining a teacher network and a student network, wherein the model structures of the teacher network and the student network are the same as the model structure of the fingerprint image segmentation network; obtaining a training data set, wherein the training data set includes on-site fingerprint images and weak labels corresponding to the on-site fingerprint images; training the teacher network and the student network based on the training data set; after confirming that the training is completed based on the loss functions corresponding to the teacher network and the student network, determining that the student network after the training is the fingerprint image segmentation network.

[0010] Optionally, the step of obtaining a training data set includes: obtaining an on-site fingerprint image and a standard fingerprint image corresponding to the on-site fingerprint image, wherein the on-site fingerprint image is the fingerprint image to be identified; matching the on-site fingerprint image and the standard fingerprint image, and generating a weak label for the on-site fingerprint image based on the matching result.

[0011] Optionally, the step of training the teacher network and the student network based on the training data set includes: determining the moving average coefficient; in each training process, training the teacher network based on the weak label, and obtaining the confidence map output by the teacher network, as well as the first teacher network parameters most recently updated by the teacher network before this training process; training the student network based on the confidence map output by the teacher network, and obtaining the student network parameters of the trained student network; determining the second teacher network parameters based on the first teacher network parameters, the student network parameters and the moving average coefficient, and updating the network parameters of the teacher network to the second teacher network parameters.

[0012] Optionally, the loss function consists of a first loss function corresponding to the teacher network and a second loss function corresponding to the student network, wherein the loss function type of the first loss function is a cross entropy loss function, and the loss function type of the second loss function is a consistency loss function.

[0013] According to another aspect of an embodiment of the present application, a fingerprint image segmentation device is also provided, including: a first processing module, used to determine a fingerprint representation matrix of a fingerprint image to be processed, wherein the fingerprint representation matrix is ​​composed of fingerprint representation vectors determined based on the pixel values ​​of the pixels in the fingerprint image to be processed; a second processing module, used to input the fingerprint representation matrix into a fingerprint image segmentation network, and obtain a confidence map output by the fingerprint image segmentation network, wherein the confidence map is used to indicate the probability that each pixel in the fingerprint image to be processed belongs to a fingerprint area, and the fingerprint image segmentation network includes multiple sequentially connected attention blocks, and a target multi-layer perceptron connected to the last attention block of the multiple sequentially connected attention blocks; a third processing module, used to generate a segmentation mask of the fingerprint image to be processed based on the confidence map, wherein the segmentation mask is used to segment the fingerprint image area and the non-fingerprint image area in the fingerprint image to be processed.

[0014] According to another aspect of an embodiment of the present application, a non-volatile storage medium is provided. The non-volatile storage medium stores a program, wherein when the program runs, the device where the non-volatile storage medium is located is controlled to execute the fingerprint image segmentation method.

[0015] According to another aspect of an embodiment of the present application, an electronic device is provided, including: a memory and a processor, wherein the processor is configured to run a program stored in the memory, wherein the fingerprint image segmentation method is executed when the program is run.

[0016] In an embodiment of the present application, a fingerprint representation matrix of a fingerprint image to be processed is determined, wherein the fingerprint representation matrix is ​​composed of fingerprint representation vectors determined based on the pixel values ​​of pixels in the fingerprint image to be processed; the fingerprint representation matrix is ​​input into a fingerprint image segmentation network, and a confidence map output by the fingerprint image segmentation network is obtained, wherein the confidence map is used to indicate the probability that each pixel in the fingerprint image to be processed belongs to the fingerprint area, and the fingerprint image segmentation network includes a plurality of sequentially connected attention blocks and a target multi-layer perceptron connected to the last attention block in the plurality of sequentially connected attention blocks; based on the confidence map, a segmentation mask of the fingerprint image to be processed is generated , wherein the segmentation mask is used to segment the fingerprint image area and the non-fingerprint image area in the fingerprint image to be processed. By adopting a fingerprint image segmentation network connected by multiple attention blocks to obtain the confidence map of the fingerprint image to be processed, and determining the segmentation mask of the fingerprint image to be processed based on the confidence map, the purpose of accurately extracting the image features of the fingerprint image area in the fingerprint image to be processed is achieved, thereby achieving the technical effect of accurately segmenting low-quality on-site fingerprint images, and further solving the technical problem of poor accuracy and robustness when using threshold segmentation and edge detection to segment on-site fingerprint images due to the poor image quality of the on-site fingerprint images. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0018] Figure 1 1 is a schematic diagram of the structure of a computer terminal (mobile terminal) provided according to an embodiment of the present application;

[0019] Figure 2 This is a schematic diagram of a method flow of a fingerprint image segmentation method provided in an embodiment of the present application;

[0020] Figure 3 This is a schematic diagram of the structure of a fingerprint image segmentation network provided according to an embodiment of the present application;

[0021] Figure 4 is a structural diagram of a self-attention mechanism module provided according to an embodiment of the present application;

[0022] Figure 5 is a schematic diagram of a training process of a fingerprint image segmentation network provided according to an embodiment of the present application;

[0023] Figure 6 This is a schematic diagram of a segmentation result of segmenting a high-quality fingerprint image using the method and related technologies provided in this application according to an embodiment of the application;

[0024] Figure 7 This is a schematic diagram of a segmentation result of a medium-quality fingerprint image segmented using the method and related technologies provided in this application according to an embodiment of the application;

[0025] Figure 8 This is a schematic diagram of a segmentation result of segmenting a low-quality fingerprint image using the method and related technologies provided in this application according to an embodiment of the application;

[0026] Figure 9 It is a structural schematic diagram of a fingerprint image segmentation device provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0027] In order to enable those skilled in the art to better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.

[0028] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in a sequence other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0029] In order to better understand the embodiments of the present application, the technical terms involved in the embodiments of the present application are explained as follows:

[0030] On-site fingerprints: On-site fingerprints refer to fingerprint images collected directly using on-site collection equipment. They are typically performed in scenarios requiring higher security, such as police investigations, crime detection, and forensic identification. On-site fingerprint collection requires specialized fingerprint collection equipment whose primary function is to capture, process, and analyze fingerprint images. On-site fingerprint collection is challenging and requires certain technical knowledge and experience to ensure effective collection and identification. Furthermore, due to the complex environment in which on-site fingerprints are collected, the image quality of on-site fingerprints is poor, often exhibiting low contrast, high noise, and complex lighting conditions. This makes it difficult to accurately extract fingerprint features from on-site fingerprints. Furthermore, in many cases, on-site fingerprints are only partially visible and may overlap with other features in the image, such as smudges or dust, making it even more difficult to distinguish the fingerprint area within the on-site fingerprint image.

[0031] Archival fingerprints: Archival fingerprints are existing fingerprint images, typically captured by a fingerprint capture device or other source and stored in a database. These fingerprints are often used for authentication and verification, such as verifying the identities of employees or members within police departments, government agencies, businesses, and academic institutions. The process of collecting and storing archival fingerprints is relatively simple and can be performed using modern fingerprint capture devices or digital cameras.

[0032] Teacher-student framework: The teacher-student framework is a semi-supervised or unsupervised learning method that aims to improve the generalization and accuracy of a model by leveraging both labeled and unlabeled data. In this framework, a teacher model is guided by an exponential moving average of the student model's previous iterative training on the dataset. The student model is then trained using both labeled and unlabeled data to improve its accuracy and generalization.

[0033] On-site fingerprint segmentation plays a crucial role in fingerprint recognition systems and has been widely used in criminal investigations, forensics, and security systems. On-site fingerprint segmentation involves extracting the fingerprint region from the on-site fingerprint image and removing background noise, thereby enhancing the visibility of the fingerprint's ridge and valley structures during subsequent processing and comparison. However, fingerprint images collected on-site often suffer from poor image quality due to environmental factors, including low contrast, high noise, and complex lighting conditions. Furthermore, in many cases, only a portion of the fingerprint is visible and may overlap with other features in the image, making it difficult to distinguish the fingerprint region.

[0034] In order to solve this problem, the related art provides a variety of segmentation methods for on-site fingerprint images, including image segmentation methods based on threshold and edge detection, region-based image segmentation methods, and deep learning-based image segmentation methods. Among them, the threshold-based image segmentation method is to separate the fingerprint and background by setting a threshold, the edge-based image segmentation method is to separate the fingerprint and background by detecting the edges of the fingerprint's ridge and valley structure, and the region-based method is to use various features (such as texture features and color features) to divide the image into regions of interest for segmentation, thereby achieving the separation of fingerprint and background. However, these methods usually require staff to manually set relevant parameters, resulting in problems in both accuracy and robustness.

[0035] Fingerprint image segmentation methods based on deep learning models have great potential for in-field fingerprint segmentation because they can learn complex features and patterns from data. Convolutional neural networks (CNNs), in particular, have been widely used in this field. However, while deep learning models themselves have great potential, the sensitivity and complexity of in-field fingerprint image data make it difficult to obtain large amounts of annotated data to supervise deep learning training. Furthermore, the training data obtained through manual annotation is severely insufficient compared to the hundreds of thousands of annotated samples required for actual matching or authentication tasks. Therefore, due to the limited training data, it is difficult to fully realize the performance of convolutional neural networks in in-field fingerprint segmentation.

[0036] In order to obtain sufficient training data, the related art provides a method of integrating various traditional fingerprint segmentation methods based on on-site fingerprint enhancement networks such as FingerNet to generate low-quality semantic segmentation labels for on-site fingerprints, thereby obtaining a large amount of training data. However, the large number of low-quality segmentation labels generated by this method cannot be directly used to train the segmentation network. If these low-quality segmentation labels are directly used, the segmentation network will learn a large amount of erroneous prior knowledge. In summary, although convolutional neural networks have great potential in the field of fingerprint segmentation, they are limited by the training data set and training methods. Therefore, the related art cannot directly apply convolutional neural networks to the field of image segmentation.

[0037] In order to solve the above problems, relevant solutions are provided in the embodiments of the present application, which are described in detail below.

[0038] According to an embodiment of the present application, a method embodiment of a fingerprint image segmentation method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0039] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 FIG1 shows a hardware structure block diagram of a computer terminal (or mobile device) for implementing a fingerprint image segmentation method. Figure 1 As shown, the computer terminal 10 (or mobile device 10) may include one or more (illustrated as 102a, 102b, ..., 102n) processors 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission module 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. It will be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components than shown, or with Figure 1 Different configurations shown.

[0040] It should be noted that the one or more processors 102 and / or other data processing circuits described above may generally be referred to herein as "data processing circuitry". The data processing circuitry may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuitry may be a single independent processing module, or may be incorporated in whole or in part into any of the other components of the computer terminal 10 (or mobile device). As described in the embodiments of the present application, the data processing circuitry serves as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0041] The memory 104 can be used to store software programs and modules for application software, such as the program instructions / data storage device corresponding to the fingerprint image segmentation method in the embodiments of the present application. The processor 102 executes the software programs and modules stored in the memory 104 to perform various functional applications and data processing, thereby implementing the above-mentioned fingerprint image segmentation method. The memory 104 may include high-speed random access memory and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories may be connected to the computer terminal 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0042] The transmission device 106 is configured to receive or transmit data via a network. A specific example of the aforementioned network may include a wireless network provided by the communications provider of the computer terminal 10. In one embodiment, the transmission device 106 includes a network interface controller (NIC), which can be connected to other network devices via a base station to enable communication with the Internet. In another embodiment, the transmission device 106 may be a radio frequency (RF) module, which is configured to communicate with the Internet wirelessly.

[0043] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 (or mobile device).

[0044] In the above operating environment, the embodiment of the present application provides a fingerprint image segmentation method, such as Figure 2 As shown, the method includes the following steps:

[0045] Step S202, determining a fingerprint representation matrix of the fingerprint image to be processed, wherein the fingerprint representation matrix is ​​composed of fingerprint representation vectors determined according to pixel values ​​of pixels in the fingerprint image to be processed;

[0046] In the technical solution provided in step S202, the step of determining the fingerprint representation matrix of the fingerprint image to be processed includes: dividing the fingerprint image to be processed into multiple image blocks of the same size, wherein no overlapping area exists between any two image blocks of the multiple image blocks of the same size; determining a fingerprint representation vector corresponding to each image block based on the pixel values ​​of the pixels in each image block, wherein the dimension of the fingerprint representation vector is the same as the number of pixels in the image block; and combining the fingerprint representation vectors corresponding to each image block to obtain a fingerprint representation matrix.

[0047] For example, assuming the size of the fingerprint image to be processed is 256×256 and the size of the image block is 32×32, the dimension of the fingerprint representation vector corresponding to each image block is 1024, the number of image blocks is 256, and the size of the above fingerprint representation matrix is ​​1024×256. After the above fingerprint representation matrix is ​​input into the fingerprint image segmentation network, after processing by each attention block, the number of vectors in the fingerprint representation matrix increases, while the dimension of each vector decreases.

[0048] In addition, in the embodiment of the present application, the fingerprint image segmentation network is also called FingerSTR.

[0049] Step S204: inputting the fingerprint representation matrix into a fingerprint image segmentation network and obtaining a confidence map output by the fingerprint image segmentation network, wherein the confidence map is used to indicate the probability that each pixel in the fingerprint image to be processed belongs to the fingerprint region. The fingerprint image segmentation network includes a plurality of sequentially connected attention blocks and a target multi-layer perceptron connected to the last attention block of the plurality of sequentially connected attention blocks.

[0050] In the solution provided in step S204, Figure 3 As shown, the multiple sequentially connected attention blocks include a first attention block, a second attention block, a third attention block, and a fourth attention block connected in sequence, wherein the number of attention heads of the first attention block is 2, representing a dimension of 64, the number of attention heads of the second attention block is 2, representing a dimension of 128, the number of attention heads of the third attention block is 4, representing a dimension of 256, and the number of attention heads of the fourth attention block is 4, representing a dimension of 512. The above-mentioned first attention block, second attention block, third attention block, and fourth attention block are all transformer network structures.

[0051] As an optional implementation, Figure 3As can be seen in the figure, the first attention block includes a linear embedding layer (Linear Embedding), and two feature extraction layers consisting of a self-attention mechanism module and a multi-layer perceptron (MLP). The second, third, and fourth attention blocks each include a downsampling layer (PatchMerging), as well as two feature extraction layers consisting of a self-attention mechanism module and a multi-layer perceptron. The fourth attention block is connected to the target multi-layer perceptron. It should be noted that different attention blocks are connected and downsampled through a single fully connected layer. Specifically, the size of the image features output by the first attention block is the same as the original image, the width and height of the image features output by the second attention block are both 1 / 2 of the original image size, the width and height of the image features output by the third attention block are both 1 / 4 of the original image size, and the width and height of the image features output by the fourth attention block are both 1 / 8 of the original image size. In addition, the downsampling process in the above-mentioned multiple attention blocks can also be implemented by the MLP layer.

[0052] Specifically, the structure of the self-attention mechanism module is as follows: Figure 4 As shown, it includes position encoding layer (PositionEmbedding), multi-head attention mechanism layer (Multi-Head Attention), residual and normalization layer (Add&Norm), multi-layer perceptron (MLP) and normalization layer (Layer Norm).

[0053] In the FingerSTR provided in the embodiments of the present application, multiple sequentially connected attention blocks are used to generate multi-level features of the fingerprint image to be processed. Furthermore, because the encoding layer in FingerSTR is composed of multiple attention blocks with a Transformer structure, and the Transformer encoder has a global receptive field, FingerSTR does not require a decoding layer that is symmetrical to the encoding layer structure, as in the convolutional segmentation network used in related art. Instead, the confidence map and segmentation mask can be obtained by simply merging the image features output by the fourth attention module through the target multi-layer perceptron.

[0054] In some embodiments of the present application, Figure 5 As shown, the fingerprint image segmentation network is trained in the following manner: determining a teacher network and a student network, wherein the model structures of the teacher network and the student network are the same as the model structure of the fingerprint image segmentation network; obtaining a training data set, wherein the training data set includes on-site fingerprint images and weak labels corresponding to the on-site fingerprint images; training the teacher network and the student network based on the training data set; after confirming that the training is completed based on the loss functions corresponding to the teacher network and the student network, determining that the student network after the training is the fingerprint image segmentation network.

[0055] Specifically, the step of obtaining a training data set includes: obtaining an on-site fingerprint image and a standard fingerprint image corresponding to the on-site fingerprint image, wherein the on-site fingerprint image is the fingerprint image to be identified; matching the on-site fingerprint image and the standard fingerprint image, and generating a weak label of the on-site fingerprint image based on the matching result. Figure 5 As can be seen from FIG, the step of matching the on-site fingerprint image and the standard fingerprint image to determine the weak label of the on-site fingerprint image can be performed by FingerNet, and the above-mentioned standard fingerprint image can be an archive fingerprint.

[0056] As an optional implementation, Figure 5 It can be seen that the EMA (exponential moving average) method can be used to update the parameters of the teacher network during the training process. Specifically, the steps of training the teacher network and the student network based on the training data set include: determining the moving average coefficient; in each training process, training the teacher network based on the weak labels, and obtaining the confidence map output by the teacher network, as well as the first teacher network parameters most recently updated by the teacher network before this training process; training the student network based on the confidence map output by the teacher network, and obtaining the student network parameters of the trained student network; determining the second teacher network parameters based on the first teacher network parameters, the student network parameters and the moving average coefficient, and updating the network parameters of the teacher network to the second teacher network parameters.

[0057] Specifically, in the teacher-student model, the exponential moving average method can be used to smooth the unlabeled data, thereby improving the robustness and generalization of both models. During the training process, the student network can update its own weights and the weights of the teacher network through moving average, thereby effectively improving the generalization performance of the teacher network and the student network on unlabeled data. The specific formula is as follows:

[0058] θ′ t =αθ′ t-1 +(1-α)θ t

[0059] In the above formula, θ′ t represents the model parameters of the teacher network at time t, θ t represents the model parameters of the student network at time t, θ′ t-1 It represents the model parameters of the teacher network at time t-1, and α is the moving average coefficient, which can be set by yourself, for example, it can be set to 0.9999.

[0060] The above loss function consists of a first loss function corresponding to the teacher network and a second loss function corresponding to the student network, wherein the loss function type of the first loss function is a cross entropy loss function, and the loss function type of the second loss function is a consistency loss function.

[0061] Specifically, the first loss function can be used to constrain the weak labels spoken by the teacher network and FingerNet. The formula of the first loss function is as follows:

[0062]

[0063] In the above formula, i represents the i-th pixel, N is the total number of pixels, represents the foreground-background label of the i-th pixel, L e (f) represents the first loss function, x i represents the pixel value of the i-th pixel, f is the predicted segmentation result, that is, the predicted label corresponding to any pixel. The value of the predicted label is 0 or 1, 0 means that the corresponding pixel belongs to the background area, and 1 means that the corresponding pixel belongs to the fingerprint area.

[0064] Since the weak labels output by FingerNet are used in the training of the teacher network, the segmentation predictions produced by the teacher network may be unreliable. To solve this problem, confidence estimates can be used to gradually guide the student model towards more reliable prediction constraints. Specifically, for a batch of training fingerprint images, the teacher model generates a segmentation prediction for each pixel and a corresponding confidence map. The training objective of the student model is then optimized by a selective consistency loss (that is, the second loss function), which only emphasizes those pixels that have high confidence according to the teacher model output. The second loss function can be denoted as L c , this loss function can utilize the pixel-level mean square error (MSE) loss between the teacher and student models. Specifically, L c It is designed to run only on the most reliable and accurate predictions, so only these high-confidence targets are focused on during training. The formula for the second loss function is as follows:

[0065]

[0066] In the above formula, II is the indicator function, f′ i is the prediction result output by the student network, f i is the prediction result output by the teacher network, p i Represents the confidence score of the i-th pixel, and H is the threshold for selecting the most reliable target. By introducing text consistency loss during training, both the student model and the teacher model can benefit from learning more reliable knowledge during training, thereby reducing the overall uncertainty of the model.

[0067] The final loss function formula is as follows:

[0068] L all =λLc +L e

[0069] Specifically, in the initial stage of training, the above λ may be set to 1 and adjusted to 0 in the subsequent training process.

[0070] In summary, during each training phase of a fingerprint segmentation network using a teacher-student model, the training fingerprint image is input into the teacher network, the student network, and FingerNet to obtain weak labels for the training fingerprint image. Loss function values ​​for the teacher network and the student network are then calculated, yielding the overall model loss function value. When calculating the teacher network's loss function value, the inputs are the teacher network's predicted output and weak labels. When calculating the student network's loss function value, the inputs are the predicted outputs of both the student and teacher networks. Furthermore, when determining the model's loss function value based on the teacher and student network's loss function values, if the current iteration count is less than half of the preset total number of iterations, the model's loss function value is equal to the sum of the teacher and student network's loss function values. If the current iteration count is greater than half of the preset total number of iterations, the model's loss function value is equal to the student network's loss function value. An EMA (Extended Mean Estimation) approach is then used to determine the teacher network's model parameters for the current round based on the student network's model parameters after the current training round and the teacher network's model parameters from the previous training round.

[0071] Step S206 : generating a segmentation mask of the fingerprint image to be processed according to the confidence map, wherein the segmentation mask is used to segment the fingerprint image area and the non-fingerprint image area in the fingerprint image to be processed.

[0072] In order to further demonstrate the advantages of the fingerprint image segmentation method provided by this application and FingerSTR over related technologies in segmenting on-site fingerprints, the following comparison process is also provided in the examples of this application. Specifically, in the examples of this application, 20K on-site fingerprints were collected from crime scenes, etc. as a training data set. The image of each on-site fingerprint is 512×512 pixels, with a resolution of 500 pixels per inch, and there are no manually marked fingerprints. The training data set is then used to train the models in FingerSTR and related technologies, and after the training is completed, the trained model is verified using the NIST SD27 data set. The data set includes 258 on-site fingerprints with expert labeling and segmentation. The size of each fingerprint is 768×800 pixels, with a resolution of 500 pixels per inch.

[0073] When evaluating the segmentation results of each model, the MDR (Missed Detection Rate) and FDR (False Detection Rate) indicators can be used to evaluate the segmentation performance of each model. The lower the value of these indicators, the more efficient the model can be considered. Assuming that A and B represent two groups of pixels in the predicted mask and the actual mask respectively, the formulas for MDR and FDR are as follows:

[0074]

[0075]

[0076] In addition, the interaction ratio (IOU) can be used to evaluate the segmentation performance of the model, and the number of floating-point operations (GFLOP) and frames per second (FPS) can be used to measure the model efficiency.

[0077] It should be noted that, in the stage of model training, data enhancement technology is used in the embodiment of the present application, and random rotation (-15°, 15°), random vertical and horizontal translation are performed in the training stage. Finally, for the high-quality on-site fingerprint image (good), the prediction results of each model are as follows: Figure 6 As shown; for the medium-quality on-site fingerprint image (bad), the prediction results of each model are as follows Figure 7 As shown; for low-quality on-site fingerprint images (ugly), the prediction results of each model are as follows Figure 8 shown. Figure 6-Figure 8 The shaded part in the figure represents the segmentation mask output by each model. It can be seen that the FingerSTR provided by the embodiment of the present application performs significantly better than the models in the related art in the bad and ugly groups.

[0078] Specifically, compared to the most advanced SegFinNet in the related art, the FingerSTR provided by this application improves the IOU indicator by approximately 11.22%, and reduces the MDR and FDR indicators by approximately -1.42% and -12.42%, respectively (the lower the indicator, the higher the segmentation accuracy). It should be noted that compared to the models in the related art, the FingerNet provided by this application does not require annotated segmentation labels.

[0079] By adopting a fingerprint representation matrix of a fingerprint image to be processed, wherein the fingerprint representation matrix is ​​composed of fingerprint representation vectors determined according to pixel values ​​of pixel points in the fingerprint image to be processed; inputting the fingerprint representation matrix into a fingerprint image segmentation network, and obtaining a confidence map output by the fingerprint image segmentation network, wherein the confidence map is used to indicate the probability that each pixel in the fingerprint image to be processed belongs to the fingerprint area, the fingerprint image segmentation network includes a plurality of sequentially connected attention blocks, and a target multi-layer perceptron connected to the last attention block of the plurality of sequentially connected attention blocks; generating a segmentation mask of the fingerprint image to be processed according to the confidence map, wherein The segmentation mask is used to segment the fingerprint image area and the non-fingerprint image area in the fingerprint image to be processed. The confidence map of the fingerprint image to be processed is obtained by adopting a fingerprint image segmentation network connected by multiple attention blocks, and the segmentation mask of the fingerprint image to be processed is determined according to the confidence map, thereby achieving the purpose of accurately extracting the image features of the fingerprint image area in the fingerprint image to be processed, thereby achieving the technical effect of accurately segmenting low-quality on-site fingerprint images, and then solving the technical problem of poor accuracy and robustness when using threshold segmentation and edge detection to segment on-site fingerprint images due to the poor image quality of the on-site fingerprint images.

[0080] The present invention provides a fingerprint image segmentation device. Figure 9 This is a schematic diagram of the structure of the fingerprint image segmentation device provided in the embodiment of the present application. Figure 9 As shown, the device includes: a first processing module 90, used to determine a fingerprint representation matrix of a fingerprint image to be processed, wherein the fingerprint representation matrix is ​​composed of fingerprint representation vectors determined based on pixel values ​​of pixels in the fingerprint image to be processed; a second processing module 92, used to input the fingerprint representation matrix into a fingerprint image segmentation network, and obtain a confidence map output by the fingerprint image segmentation network, wherein the confidence map is used to indicate the probability that each pixel in the fingerprint image to be processed belongs to a fingerprint area, and the fingerprint image segmentation network includes a plurality of sequentially connected attention blocks, and a target multi-layer perceptron connected to the last attention block in the plurality of sequentially connected attention blocks; a third processing module 94, used to generate a segmentation mask of the fingerprint image to be processed based on the confidence map, wherein the segmentation mask is used to segment the fingerprint image area and the non-fingerprint image area in the fingerprint image to be processed.

[0081] In some embodiments of the present application, the step of the first processing module 90 determining the fingerprint representation matrix of the fingerprint image to be processed includes: dividing the fingerprint image to be processed into multiple image blocks of the same size, wherein there is no overlapping area between any two image blocks of the multiple image blocks of the same size; determining the fingerprint representation vector corresponding to each image block based on the pixel values ​​of the pixels in each image block, wherein the dimension of the fingerprint representation vector is the same as the number of pixels in the image block; and combining the fingerprint representation vectors corresponding to each image block to obtain a fingerprint representation matrix.

[0082] In some embodiments of the present application, a plurality of sequentially connected attention blocks include a first attention block, a second attention block, a third attention block and a fourth attention block connected in sequence, wherein the number of attention heads of the first attention block is 2, representing a dimension of 64, the number of attention heads of the second attention block is 2, representing a dimension of 128, the number of attention heads of the third attention block is 4, representing a dimension of 256, and the number of attention heads of the fourth attention block is 4, representing a dimension of 512.

[0083] In some embodiments of the present application, the first attention block includes a linear embedding layer and two feature extraction layers consisting of a self-attention mechanism module and a multilayer perceptron. The second attention block, the third attention block, and the fourth attention block each include a downsampling layer and two feature extraction layers consisting of a self-attention mechanism module and a multilayer perceptron. The fourth attention block is connected to the target multilayer perceptron.

[0084] In some embodiments of the present application, a fingerprint image segmentation network is trained in the following manner: determining a teacher network and a student network, wherein the model structures of the teacher network and the student network are the same as the model structure of the fingerprint image segmentation network; obtaining a training data set, wherein the training data set includes on-site fingerprint images and weak labels corresponding to the on-site fingerprint images; training the teacher network and the student network based on the training data set; after confirming that the training is completed based on the loss functions corresponding to the teacher network and the student network, determining that the student network after the training is completed is a fingerprint image segmentation network.

[0085] In some embodiments of the present application, the step of obtaining a training data set includes: obtaining an on-site fingerprint image and a standard fingerprint image corresponding to the on-site fingerprint image, wherein the on-site fingerprint image is the fingerprint image to be identified; matching the on-site fingerprint image and the standard fingerprint image, and generating a weak label for the on-site fingerprint image based on the matching result.

[0086] In some embodiments of the present application, the steps of the fingerprint image segmentation device training the teacher network and the student network based on the training data set include: determining the moving average coefficient; in each training process, training the teacher network based on the weak label, and obtaining the confidence map output by the teacher network, as well as the first teacher network parameters most recently updated by the teacher network before this training process; training the student network based on the confidence map output by the teacher network, and obtaining the student network parameters of the trained student network; determining the second teacher network parameters based on the first teacher network parameters, the student network parameters and the moving average coefficient, and updating the network parameters of the teacher network to the second teacher network parameters.

[0087] In some embodiments of the present application, the loss function consists of a first loss function corresponding to the teacher network and a second loss function corresponding to the student network, wherein the loss function type of the first loss function is a cross entropy loss function, and the loss function type of the second loss function is a consistency loss function.

[0088] It should be noted that the various modules in the above-mentioned fingerprint image segmentation device can be program modules (for example, a set of program instructions that implement a certain specific function) or hardware modules. For the latter, it can be expressed in the following forms, but is not limited to this: the expression form of each of the above-mentioned modules is a processor, or the functions of each of the above-mentioned modules are implemented by a processor.

[0089] According to an embodiment of the present application, a non-volatile storage medium is also provided, in which a program is stored, wherein when the program is running, the device where the non-volatile storage medium is located is controlled to perform the following fingerprint image segmentation method: determining a fingerprint representation matrix of the fingerprint image to be processed, wherein the fingerprint representation matrix is ​​composed of fingerprint representation vectors determined based on the pixel values ​​of the pixels in the fingerprint image to be processed; inputting the fingerprint representation matrix into a fingerprint image segmentation network, and obtaining a confidence map output by the fingerprint image segmentation network, wherein the confidence map is used to indicate the probability that each pixel in the fingerprint image to be processed belongs to a fingerprint area, and the fingerprint image segmentation network includes multiple sequentially connected attention blocks, and a target multi-layer perceptron connected to the last attention block of the multiple sequentially connected attention blocks; generating a segmentation mask of the fingerprint image to be processed based on the confidence map, wherein the segmentation mask is used to segment the fingerprint image area and the non-fingerprint image area in the fingerprint image to be processed.

[0090] According to an embodiment of the present application, an electronic device is also provided, which includes a processor and a memory, the processor being used to run a program stored in the memory, wherein the program executes the following fingerprint image segmentation method when it is run: determining a fingerprint representation matrix of a fingerprint image to be processed, wherein the fingerprint representation matrix is ​​composed of fingerprint representation vectors determined based on the pixel values ​​of pixels in the fingerprint image to be processed; inputting the fingerprint representation matrix into a fingerprint image segmentation network, and obtaining a confidence map output by the fingerprint image segmentation network, wherein the confidence map is used to indicate the probability that each pixel in the fingerprint image to be processed belongs to a fingerprint area, the fingerprint image segmentation network including a plurality of sequentially connected attention blocks, and a target multilayer perceptron connected to the last attention block of the plurality of sequentially connected attention blocks; generating a segmentation mask of the fingerprint image to be processed based on the confidence map, wherein the segmentation mask is used to segment the fingerprint image area and the non-fingerprint image area in the fingerprint image to be processed.

[0091] According to an embodiment of the present application, a computer program product is also provided. The computer program product can be run by a processor or electronic device and, when run, performs the following fingerprint image segmentation method: determining a fingerprint representation matrix of a fingerprint image to be processed, wherein the fingerprint representation matrix is ​​composed of fingerprint representation vectors determined based on pixel values ​​of pixels in the fingerprint image to be processed; inputting the fingerprint representation matrix into a fingerprint image segmentation network, and obtaining a confidence map output by the fingerprint image segmentation network, wherein the confidence map is used to indicate the probability that each pixel in the fingerprint image to be processed belongs to a fingerprint region, the fingerprint image segmentation network including a plurality of sequentially connected attention blocks and a target multilayer perceptron connected to the last attention block in the plurality of sequentially connected attention blocks; generating a segmentation mask for the fingerprint image to be processed based on the confidence map, wherein the segmentation mask is used to segment the fingerprint image to be processed into a fingerprint image region and a non-fingerprint image region.

[0092] In the above embodiments of the present application, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0093] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only exemplary. For example, the division of the units can be a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0094] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0095] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0096] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the relevant technology or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0097] The above is only a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A fingerprint image segmentation method, characterized in that: include: Determining a fingerprint representation matrix of the fingerprint image to be processed, wherein the fingerprint representation matrix is ​​composed of fingerprint representation vectors determined according to pixel values ​​of pixels in the fingerprint image to be processed; The fingerprint representation matrix is ​​input into a fingerprint image segmentation network, and a confidence map output by the fingerprint image segmentation network is obtained, wherein the confidence map is used to indicate the probability that each pixel in the fingerprint image to be processed belongs to the fingerprint area, and the fingerprint image segmentation network includes a plurality of sequentially connected attention blocks, and a target multilayer perceptron connected to the last attention block of the plurality of sequentially connected attention blocks, wherein the plurality of sequentially connected attention blocks include a first attention block, a second attention block, a third attention block and a fourth attention block connected in sequence, wherein the number of attention heads of the first attention block is 2, and the representation dimension is 64, the number of attention heads of the second attention block is 2, indicating a dimension of 128, the number of attention heads of the third attention block is 4, indicating a dimension of 256, the number of attention heads of the fourth attention block is 4, indicating a dimension of 512, the first attention block includes a linear embedding layer, and two feature extraction layers consisting of a self-attention mechanism module and a multilayer perceptron, the second attention block, the third attention block, and the fourth attention block each include a downsampling layer, and two feature extraction layers consisting of a self-attention mechanism module and a multilayer perceptron, and the fourth attention block is connected to the target multilayer perceptron; Generating a segmentation mask of the fingerprint image to be processed according to the confidence map, wherein the segmentation mask is used to segment the fingerprint image area and the non-fingerprint image area in the fingerprint image to be processed; The step of determining the fingerprint representation matrix of the fingerprint image to be processed includes: dividing the fingerprint image to be processed into multiple image blocks of the same size, wherein no overlapping area exists between any two image blocks of the multiple image blocks of the same size; determining the fingerprint representation vector corresponding to each image block based on the pixel values ​​of the pixels in each image block, wherein the dimension of the fingerprint representation vector is the same as the number of pixels in the image block; and combining the fingerprint representation vectors corresponding to each image block to obtain the fingerprint representation matrix.

2. The fingerprint image segmentation method according to any one of claim 1, characterized in that: The fingerprint image segmentation network is trained in the following way: Determining a teacher network and a student network, wherein the model structures of the teacher network and the student network are the same as the model structure of the fingerprint image segmentation network; Acquire a training data set, wherein the training data set includes an on-site fingerprint image and a weak label corresponding to the on-site fingerprint image; Training the teacher network and the student network based on the training data set; After confirming that the training is completed according to the loss functions corresponding to the teacher network and the student network, the student network after training is determined to be the fingerprint image segmentation network.

3. The fingerprint image segmentation method according to claim 2, characterized in that: The step of obtaining a training data set includes: Acquire an on-site fingerprint image and a standard fingerprint image corresponding to the on-site fingerprint image, wherein the on-site fingerprint image is the fingerprint image to be identified; The on-site fingerprint image and the standard fingerprint image are matched, and a weak label of the on-site fingerprint image is generated according to the matching result.

4. The fingerprint image segmentation method according to claim 2, characterized in that: The step of training the teacher network and the student network according to the training data set includes: Determine the moving average coefficient; In each training process, the teacher network is trained according to the weak label, and a confidence map output by the teacher network and a first teacher network parameter most recently updated by the teacher network before the current training process are obtained; Training the student network according to the confidence map output by the teacher network, and obtaining student network parameters of the trained student network; Determine second teacher network parameters based on the first teacher network parameters, the student network parameters and the moving average coefficient, and update the network parameters of the teacher network to the second teacher network parameters.

5. The fingerprint image segmentation method according to claim 2, characterized in that: The loss function consists of a first loss function corresponding to the teacher network and a second loss function corresponding to the student network, wherein the loss function type of the first loss function is a cross entropy loss function, and the loss function type of the second loss function is a consistency loss function.

6. A fingerprint image segmentation device, characterized in that: include: A first processing module is configured to determine a fingerprint representation matrix of a fingerprint image to be processed, wherein the fingerprint representation matrix is ​​composed of fingerprint representation vectors determined according to pixel values ​​of pixels in the fingerprint image to be processed; The second processing module is used to input the fingerprint representation matrix into the fingerprint image segmentation network and obtain the confidence map output by the fingerprint image segmentation network, wherein the confidence map is used to indicate the probability that each pixel in the fingerprint image to be processed belongs to the fingerprint area, and the fingerprint image segmentation network includes a plurality of sequentially connected attention blocks, and a target multilayer perceptron connected to the last attention block of the plurality of sequentially connected attention blocks, wherein the plurality of sequentially connected attention blocks include a first attention block, a second attention block, a third attention block and a fourth attention block connected in sequence, wherein the number of attention heads of the first attention block is 2, indicating The representation dimension is 64, the number of attention heads of the second attention block is 2, the representation dimension is 128, the number of attention heads of the third attention block is 4, the representation dimension is 256, the number of attention heads of the fourth attention block is 4, the representation dimension is 512, the first attention block includes a linear embedding layer, and two feature extraction layers consisting of a self-attention mechanism module and a multilayer perceptron, the second attention block, the third attention block and the fourth attention block each include a downsampling layer, and two feature extraction layers consisting of a self-attention mechanism module and a multilayer perceptron, and the fourth attention block is connected to the target multilayer perceptron; a third processing module, configured to generate a segmentation mask of the fingerprint image to be processed based on the confidence map, wherein the segmentation mask is used to segment the fingerprint image area and the non-fingerprint image area in the fingerprint image to be processed; The first processing module is further used to: divide the fingerprint image to be processed into multiple image blocks of the same size, wherein there is no overlapping area between any two image blocks of the multiple image blocks of the same size; determine the fingerprint representation vector corresponding to each image block based on the pixel values ​​of the pixels in each image block, wherein the dimension of the fingerprint representation vector is the same as the number of pixels in the image block; and combine the fingerprint representation vectors corresponding to each image block to obtain the fingerprint representation matrix.

7. A non-volatile storage medium, characterized in that: The non-volatile storage medium stores a program, wherein when the program is running, the device where the non-volatile storage medium is located is controlled to execute the fingerprint image segmentation method according to any one of claims 1 to 5.

8. An electronic device, characterized in that: include: A memory and a processor, wherein the processor is configured to run a program stored in the memory, wherein the fingerprint image segmentation method according to any one of claims 1 to 5 is executed when the program is run.

Images