Method and apparatus for estimating finger three-dimensional pose from planar fingerprint

Abstract

The application provides a method for estimating a three-dimensional posture of a finger from a planar fingerprint, and relates to the technical field of human-computer interaction and touch screen interaction, and the method comprises the following steps: collecting a planar fingerprint image to be tested; inputting the planar fingerprint image to be tested into a preset deep neural network model, and outputting a predicted posture angle. The application adopting the above scheme solves the technical problems of low sensor resolution and limited posture estimation accuracy of the prior art, solves the problem that the prior art needs to additionally introduce hardware, which hinders the actual application scenario, solves the problem that the prior art needs to collect a complete unfolded fingerprint, which brings difficulties to the actual landing of the scheme, and achieves the purposes of enhancing the safety and reliability of fingerprint identification application, enriching touch screen information input, improving user experience, and promoting fingerprint interactive application.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of human-computer interaction and touch screen interaction, and particularly relates to a method and device for estimating a three-dimensional posture of a finger from a planar fingerprint. BACKGROUND

[0002] In the field of human-computer interaction, touch screen input is favored by a large number of intelligent mobile devices as a simple and fast interaction method. The popularity of intelligent devices is inseparable from the iterative update of the touch screen interaction method in addition to the development of chip manufacturing, mobile phone systems and other technologies. The traditional intelligent device can sense the pressing area of the user's finger and give corresponding feedback. In the case of picture enlargement, rotation, scaling and other operations, the user needs to use at least two fingers to successfully complete the above operations. If the angle of the finger when contacting the touch screen can be accurately estimated as additional input information, the user experience will be greatly improved, the logic of part of the operation will be simplified, the application scenarios of fingerprint input will be widened, and more interesting interaction forms will be brought.

[0003] Fingerprint, as an important biometric information, has been widely used in identity authentication of smart phones, notebook computers and the like. Due to some defects in the fingerprint recognition model, criminals can easily use these loopholes to construct false fingerprints to cheat the identity recognition system, which will cause great loss to the national economy, social stability and personal information security. In face recognition, there are the same problems. Researchers in this field have invented a live body detection technology that can verify whether the user is a real live body by using a combination of actions such as blinking, opening the mouth, shaking the head, nodding, and using face key point positioning and face tracking technology. It can effectively resist common attack methods such as photos, face changing, masks, shielding and screen flipping, thereby helping users to identify fraudulent behavior. Then back to the field of fingerprint recognition, the live body detection algorithm is still lacking.

[0004] In the existing touch screen interaction, since the information accepted by the intelligent device from the sensor only includes the number and position of the pressing finger, the gestures combined therefrom are limited and cannot meet the needs of more and more interaction applications.

[0005] In the field of three-dimensional posture estimation of fingerprints, the existing technologies can be divided into the following types according to the modalities of the input fingerprints:

[0006] I. Posture estimation based on capacitive sensing type fingerprint

[0007] Capacitive sensing fingerprint is to image the difference of sensor capacitance between the case of fingerprint touching the screen and not touching the screen. It is widely used in touch screen devices due to its relatively simple principle. (Xiao et al.) extracted 42 features from the capacitive sensing fingerprint, and trained a Gaussian regression model to estimate the pitch angle and yaw angle, and achieved good accuracy on smartphones and smartwatches. However, they could not predict the roll angle, so the obtained fingerprint pose was incomplete. Moreover, due to the low resolution of such sensors, the accuracy of pose estimation is limited.

[0008] II. Pose estimation fusing other modal information

[0009] (Mayer et al.) used an externally mounted depth camera to obtain the depth information of the fingerprint, and fused prior knowledge to constrain the range of pitch angle to 0-90 degrees. (Watanabe et al.) bound a camera on the fingertip of the experimental subject, and calculated the pitch angle and yaw angle by detecting the change of light intensity of the nail cover. These schemes use modal information other than planar fingerprint to obtain the complete three-dimensional pose of the fingerprint, but need to introduce additional hardware, which hinders the practical application scenario.

[0010] III. Pose estimation based on planar pressed fingerprint

[0011] So far, there is only one scheme that directly estimates the three-dimensional pose of the finger based on the planar pressed fingerprint. (Holz and Baudisch) calculated the spatial transformation relationship between the input fingerprint and the corresponding object in the database to obtain the three-dimensional pose of the current input fingerprint, including the pitch angle, roll angle and yaw angle. This scheme uses the same sensor as the present application, but it needs to establish a complete database of expanded fingerprints corresponding to each input fingerprint, which cannot predict the planar pressed fingerprint of any new user. Collecting complete expanded fingerprints is a tedious and complicated process, which brings difficulties to the practical application of these technical solutions.

[0012] Existing three-dimensional fingerprint pose estimation technical solutions either cannot accurately estimate the complete fingerprint pose or need additional sensor devices for assistance, which are not ideal in terms of function and convenience. SUMMARY

[0013] The present application aims to at least partially solve one of the technical problems in the related art.

[0014] To this end, a first object of the present application is to provide a method for estimating a three-dimensional posture of a finger from a planar fingerprint, which solves the technical problem of low sensor resolution and limited posture estimation accuracy of existing methods, and also solves the problem of additional hardware required by existing methods, which hinders practical application scenarios, and further solves the problem of the need for complete spread fingerprints in existing methods, which brings difficulties to the actual landing of the scheme, and achieves the purpose of enhancing the security and reliability of fingerprint recognition applications, enriching touch screen information input, improving user experience, and promoting fingerprint interactive applications.

[0015] A second object of the present application is to provide a device for estimating a three-dimensional posture of a finger from a planar fingerprint.

[0016] A third object of the present application is to provide a non-transitory computer-readable storage medium.

[0017] To achieve the above objects, a first aspect of the present application provides a method for estimating a three-dimensional posture of a finger from a planar fingerprint, comprising: collecting a planar fingerprint image to be tested; inputting the planar fingerprint image to be tested into a preset deep neural network model to output a predicted posture angle.

[0018] Optionally, in an embodiment of the present application, the deep neural network model comprises a feature extraction backbone network, an attention mechanism module, and a three-dimensional angle prediction module, and the feature extraction backbone network comprises a convolution layer and a nonlinear activation layer.

[0019] Optionally, in an embodiment of the present application, inputting the planar fingerprint image to be tested into the preset deep neural network model to output the predicted posture angle comprises the following steps:

[0020] The planar fingerprint image to be tested is subjected to feature extraction through stacking and abstraction of the convolution layer and the nonlinear activation layer, and a fixed-dimension feature descriptor is outputted;

[0021] The attention mechanism module is used to perform attention mechanism calculation on the fixed-dimension feature descriptor, the output heat map is fused with the feature descriptor, and an optimized feature descriptor is obtained;

[0022] The value range of the pitch angle, roll angle, and yaw angle of the optimized feature descriptor is divided into several intervals, and then the three-dimensional angle prediction module is used to output the probability that the current fingerprint image belongs to each interval, and based on this probability and the midpoint value of the divided interval, a weighted final predicted posture angle is obtained.

[0023] Optionally, in an embodiment of the present application, the deep neural network model is trained, comprising the following steps:

[0024] Obtaining training data, including a database of planar fingerprint images and corresponding three-dimensional postures of fingers;

[0025] Using the training data, the deep neural network module is trained using a loss function.

[0026] Optionally, in an embodiment of the present application, obtaining the training data comprises:

[0027] Rolling the finger with the gyroscope bound on the fingerprint sensor from one side to the other side;

[0028] Collecting multiple planar fingerprint images at a fixed frequency by program control of the fingerprint collector, and the gyroscope synchronously reading out the angle;

[0029] Calculating the three-dimensional pose angle corresponding to each planar fingerprint image by the difference between the values of the two gyroscopes.

[0030] Optionally, in an embodiment of the present application, obtaining the training data comprises, synchronously obtaining the finger pose and the fingerprint image by using optical tracking technology combined with the fingerprint collector, or synthesizing the planar fingerprint image under any pose by using a three-dimensional fingerprint database constructed in advance.

[0031] Optionally, in an embodiment of the present application, the loss function is represented as:

[0032]

[0033]

[0034]

[0035] L roll / pitch / yaw = L ce + λL MSE

[0036] L all = L roll + L pitch + L yaw

[0037] Wherein, L CE is the cross-entropy loss function of the classification problem, y c is 0 or 1, y c = 1 if and only if c is the current pose angle distribution in the cth interval, p c (c ∈ [1, M]) represents the probability value of the network prediction belonging to each interval, s c represents the representative value of each angle interval, and M represents the total number of intervals, is the angle prediction value, θ is the angle true value, L roll / pitch / yaw is the loss function of a single pose angle, λ is a hyperparameter, and is taken as 1.0, L all is the total loss function of all pose angles.

[0038] To achieve the above object, the second aspect of the present application provides a device for estimating a three-dimensional posture of a finger from a planar fingerprint, comprising a data acquisition module and a prediction module, wherein

[0039] The data acquisition module is configured to acquire a planar fingerprint image to be tested.

[0040] The prediction module is configured to predict the planar fingerprint image to be tested and output a predicted posture angle.

[0041] Optionally, in an embodiment of the present application, the prediction module further comprises a training unit, and the training unit is specifically configured to:

[0042] Obtain training data, including a database of planar fingerprint images and corresponding three-dimensional postures of fingers.

[0043] Use the training data to train the deep neural network module using a loss function.

[0044] To achieve the above object, the third aspect of the present application provides a non-transitory computer-readable storage medium, when the instructions in the storage medium are executed by a processor, the method and device for estimating a three-dimensional posture of a finger from a planar fingerprint can be executed.

[0045] The method for estimating a three-dimensional posture of a finger from a planar fingerprint, the device for estimating a three-dimensional posture of a finger from a planar fingerprint and the non-transitory computer-readable storage medium provided by the embodiments of the present application solve the technical problem that the existing method has low sensor resolution and limited posture estimation accuracy, solve the problem that the existing method needs to introduce additional hardware, which hinders the actual application scenario, solve the problem that the existing method needs to collect a complete unfolded fingerprint, which brings difficulties to the actual landing of the scheme, achieve the purpose of enhancing the security and reliability of fingerprint recognition application, innovating and supplementing the interactive mode, enriching the touch screen information input, increasing the interactive interest, bridging the gap between touch screen devices and traditional physical keyboard devices while maintaining the advantages of simplicity and ease of use, improving user experience, and promoting fingerprint interactive applications.

[0046] Additional aspects and advantages of the present application will be in part apparent and in part pointed out hereinafter. BRIEF DESCRIPTION OF DRAWINGS

[0047] The above and / or additional aspects and advantages of the present application will become apparent and be readily appreciated from the following description, including the accompanying drawings, in which:

[0048] Figure 1 A flowchart of a method for estimating a three-dimensional posture of a finger from a planar fingerprint provided by the first embodiment of the present application;

[0049] Figure 2 FIG. 1 is a diagram illustrating a definition of a three-dimensional posture of a finger when the finger presses a screen according to an embodiment of the method for estimating a three-dimensional posture of a finger from a planar fingerprint of the present application;

[0050] Figure 3 FIG. 2 is a flowchart of another embodiment of the method for estimating a three-dimensional posture of a finger from a planar fingerprint of the present application;

[0051] Figure 4 FIG. 3 is a diagram of a data acquisition instrument according to an embodiment of the method for estimating a three-dimensional posture of a finger from a planar fingerprint of the present application;

[0052] Figure 5 FIG. 4 is a diagram of a neural network structure according to an embodiment of the method for estimating a three-dimensional posture of a finger from a planar fingerprint of the present application. DETAILED DESCRIPTION

[0053] Embodiments of the present application are described in detail below with reference to the attached drawings, which are meant to be exemplary and not limiting.

[0054] A method and apparatus for estimating a three-dimensional posture of a finger from a planar fingerprint according to embodiments of the present application are described below with reference to the attached drawings.

[0055] Figure 1 FIG. 1 is a diagram illustrating a definition of a three-dimensional posture of a finger when the finger presses a screen according to an embodiment of the method for estimating a three-dimensional posture of a finger from a planar fingerprint of the present application;

[0056] As shown in FIG. 1, the method for estimating a three-dimensional posture of a finger from a planar fingerprint includes the following steps: Figure 1

[0057] Step 101, acquiring a planar fingerprint image to be tested;

[0058] Step 102, inputting the planar fingerprint image to be tested into a preset deep neural network model, and outputting a predicted posture angle.

[0059] ​The method for estimating a three-dimensional posture of a finger from a planar fingerprint according to the embodiments of the present application can collect a planar fingerprint image to be tested; input the planar fingerprint image to be tested into a preset deep neural network model, and output a predicted posture angle. Thus, the technical problem of low sensor resolution and limited posture estimation accuracy of the existing method can be solved, the problem of the need for additional hardware in the existing method and the hindrance to the actual application scenario can be solved, the problem of the need for the collection of a complete expanded fingerprint in the existing method and the difficulty in the actual landing of the scheme can be solved, the safety and reliability of fingerprint recognition application are enhanced, the interactive mode is innovated and supplemented, the touch screen information input is enriched, the interactive interest is increased, the gap between the touch screen device and the traditional physical key-in device is bridged under the advantage of keeping simple and easy to use, the user experience is improved, and the purpose of promoting the fingerprint interactive application is achieved.

[0060] Further, in the embodiments of the present application, the deep neural network model comprises a feature extraction backbone network, an attention mechanism module, and a three-dimensional angle prediction module, and the feature extraction backbone network comprises a convolution layer and a nonlinear activation layer.

[0061] Further, in the embodiments of the present application, the planar fingerprint image to be tested is input into the preset deep neural network model, and a predicted posture angle is output, which comprises the following steps:

[0062] The planar fingerprint image to be tested is subjected to feature extraction through stacking and abstraction of the convolution layer and the nonlinear activation layer, and a fixed-dimension feature descriptor is output;

[0063] The attention mechanism module is used to perform attention mechanism calculation on the fixed-dimension feature descriptor, the output heat map is fused with the feature descriptor, and an optimized feature descriptor is obtained;

[0064] The value range of the pitch angle, the roll angle, and the yaw angle of the optimized feature descriptor is divided into several intervals, then the three-dimensional angle prediction module is used to output the probability that the current fingerprint image belongs to each interval, and based on the probability and the midpoint value of the divided interval, a final predicted posture angle is obtained by weighting.

[0065] In recent years, deep neural networks have achieved great success in computer vision, medical image processing, natural language processing and other fields due to their powerful representation ability and ability to mine information from large-scale data. In order to make full use of the information in the fingerprint image and improve the accuracy of posture prediction, a deep neural network model is designed. Feature extraction module: learn from the current most powerful backbone network framework, follow the backbone network of (Yin et al.), and make some modifications, make full use of the skill of multi-scale fusion, and finally output a fixed dimension feature descriptor; Attention mechanism module: in order to make the network pay more attention to the foreground area of the fingerprint image and ignore the invalid background information, combine and modify the latest attention mechanism module in (Yin et al.), the output of this module will be used as a mask to enhance the foreground area in the output of the feature extraction module; Posture angle regression module: fusion and optimization of the angle regression module proposed in (Ruiz et al.), which is the best method in head pose estimation. Assuming that there is no dependency between the three angles, the independent prediction scheme is adopted.

[0066] Further, in the embodiment of the present application, training the deep neural network model comprises the following steps:

[0067] Obtaining training data, including a database of planar fingerprint images and corresponding finger three-dimensional postures;

[0068] Using the training data, the deep neural network module is trained using a loss function.

[0069] In order to train an accurate model, a database of planar fingerprint images and corresponding finger three-dimensional postures needs to be established first. The accuracy of the deep learning model depends on the size of the database, so the data collection scheme is an important part of the design of the present application.

[0070] Further, in the embodiment of the present application, obtaining training data comprises:

[0071] Rolling the finger tied with the gyroscope on the fingerprint sensor from one side to the other side;

[0072] Collecting multiple planar fingerprint images at a fixed frequency by program control of the fingerprint collector, and the gyroscope synchronously reads out the angle;

[0073] The three-dimensional posture angle corresponding to each planar fingerprint image is calculated by the difference between the values of the two gyroscopes.

[0074] The data acquisition is performed by using a common flat fingerprint collector (for collecting flat fingerprint images) and three-axis gyroscopes (for reading three-dimensional angle data) in two aircrafts. The two gyroscopes are fixed on the fingerprint sensor and the finger to be collected, respectively, and the three-dimensional angle of the finger relative to the plane of the fingerprint sensor is obtained by the difference between the readings of the two gyroscopes.

[0075] Further, in the embodiments of the present application, the training data is obtained by using optical tracking technology combined with a fingerprint collector to synchronously obtain finger posture and fingerprint images, or by using a three-dimensional fingerprint database constructed in advance to synthesize flat fingerprint images in any posture.

[0076] Further, in the embodiments of the present application, the loss function is represented as:

[0077]

[0078]

[0079]

[0080] L roll / pitch / yaw = L ce + λL MSE

[0081] L all = L roll + L pitch + L yaw

[0082] wherein L CE is a cross-entropy loss function for a classification problem, y c is 0 or 1, y c = 1 if and only if c is the current posture angle distribution in the cth interval, p c (c ∈ [1, M]) represents a probability value predicted by the network to belong to each interval, s c represents a representative value of each angle interval, and M represents the total number of intervals, is an angle prediction value, θ is an angle true value, L roll / pitch / yaw is a loss function for a single posture angle, λ is a hyperparameter, and is taken as 1.0, L all is a total loss function for all posture angles.

[0083] Due to the differences in finger shapes of each person, ambiguity and fuzziness are prone to occur when the angle is predicted, thereby reducing the accuracy of posture estimation. In view of this situation, two loss functions are designed to assist each other to reduce the posture error. After obtaining the probability of each prediction interval by using the neural network proposed in the above step, a comparison is made with the interval of the true value distribution, and a classical cross-entropy loss function for a classification problem is used. The prediction probability pc and the midpoint s of the divided angle interval c the angle prediction value after weighted average Compare with the angle true value θ, and calculate the regression loss function using L2 norm. The final loss function of a single pose angle is the weighted sum of the above two.

[0084] Figure 2 The figure shows the definition of the three-dimensional pose of the finger when the finger presses the screen in the method for estimating the three-dimensional pose of the finger from the planar fingerprint according to the embodiments of the present application.

[0085] As shown in Figure 2 The three-dimensional pose of the finger when the finger presses the screen in the method for estimating the three-dimensional pose of the finger from the planar fingerprint includes: the yaw angle, which refers to the angle of rotation of the finger about the vertical screen plane; the pitch angle, which refers to the angle formed by the long axis of the finger and the screen plane; and the roll angle, which refers to the angle of rotation of the finger about the long axis.

[0086] Figure 3 Another flowchart of the method for estimating the three-dimensional pose of the finger from the planar fingerprint according to the embodiments of the present application is shown.

[0087] As shown in Figure 3 The method for estimating the three-dimensional pose of the finger from the planar fingerprint includes: estimating the three-dimensional pose of the finger from the planar pressing fingerprint, learning the mapping rule from the constructed data set using the deep learning method, including: establishing a database of planar fingerprint images and corresponding three-dimensional poses of the finger; collecting data using a commonly used planar fingerprint collector (for collecting planar fingerprint images) and a three-axis gyroscope (for reading three-dimensional angle data) used in two aircrafts; following the backbone network of (Yin et al.), some modifications are made, which fully utilizes the multi-scale fusion technique and outputs a fixed-dimensional feature descriptor; fusing and optimizing the angle regression module proposed in (Ruiz et al.), dividing the value range of the pitch angle, the roll angle and the yaw angle to be regressed into several intervals, using this module to output the probability that the current fingerprint image belongs to each interval, based on the probability and the midpoint value of the divided interval, weighted to obtain the final predicted pose angle; using the loss function to train the deep neural network module.

[0088] Figure 4 The figure shows the data collection instrument of the method for estimating the three-dimensional pose of the finger from the planar fingerprint according to the embodiments of the present application.

[0089] As shown in Figure 4As shown, the method for estimating the three-dimensional posture of a finger from a plane fingerprint will use a commonly used plane fingerprint collector (for collecting a plane fingerprint image) and a three-axis gyroscope (for reading three-dimensional angle data) used in two aircrafts to collect data. Two gyroscopes are fixed on the fingerprint sensor and the finger to be collected, respectively, and the three-dimensional angle of the finger relative to the plane of the fingerprint sensor is obtained by the difference between the readings of the two.

[0090] Figure 5 The neural network structure diagram of the method for estimating the three-dimensional posture of a finger from a plane fingerprint according to the embodiment of the application.

[0091] As shown, Figure 5 In order to make full use of the information in the pressed fingerprint image and improve the accuracy of posture prediction, the neural network is designed. According to its function, it can be divided into: a feature extraction backbone network, which follows the backbone network of (Yin et al.) and makes some modifications, fully utilizes the multi-scale fusion technique, and finally outputs a fixed-dimensional feature descriptor; an attention mechanism module, in order to enable the network to pay more attention to the foreground area of the fingerprint image and ignore invalid background information, the latest attention mechanism module in (Yin et al.) is combined and modified, the output of this module will be used as a mask to enhance the foreground area in the output of the feature extraction module; a three-dimensional angle prediction module, which integrates and optimizes the angle regression module proposed in (Ruiz et al.), divides the value range of the pitch angle, roll angle and yaw angle to be regressed into several intervals, and then outputs the probability that the current fingerprint image belongs to each interval using this module, and based on this probability and the midpoint value of the divided interval, the final predicted posture angle is obtained by weighting.

[0092] The second embodiment of the application proposes a device for estimating the three-dimensional posture of a finger from a plane fingerprint, which comprises a data collection module and a prediction module, wherein,

[0093] The data collection module is used to collect the plane fingerprint image to be tested.

[0094] The prediction module is used to predict the plane fingerprint image to be tested and output the predicted posture angle.

[0095] Further, in the embodiment of the application, the prediction module further comprises a training unit, and the training unit is specifically used for:

[0096] Obtaining training data, including a database of plane fingerprint images and corresponding three-dimensional postures of fingers;

[0097] Using the training data, the deep neural network module is trained using a loss function.

[0098] The device for estimating three-dimensional posture of a finger from a planar fingerprint of the embodiment comprises a data acquisition module configured to acquire a planar fingerprint image to be tested; and a prediction module configured to predict the planar fingerprint image to be tested and output a predicted posture angle. Thus, the technical problem of low sensor resolution and limited posture estimation accuracy of the prior art method can be solved, the problem of additional hardware required by the prior art method and the problem of difficulty in actual application scenarios caused by the requirement of the prior art method for collecting a complete expanded fingerprint can be solved, the safety and reliability of fingerprint recognition application can be enhanced, the interactive mode can be innovated and supplemented, the touch screen information input can be enriched, the interaction interest can be increased, the gap between the touch screen device and the traditional physical keyboard device can be bridged under the advantage of keeping simple and easy to use, the user experience can be improved, and the purpose of promoting the fingerprint interactive application can be achieved.

[0099] In order to achieve the above-mentioned embodiments, the application further provides a non-transitory computer readable storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement the method and device for estimating three-dimensional posture of a finger from a planar fingerprint.

[0100] In the description of the present specification, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the present specification, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in any one or more embodiments or examples. In addition, the person skilled in the art can combine and combine the different embodiments or examples described in the present specification and the features of the different embodiments or examples without contradiction.

[0101] In addition, the terms "first", "second" are only for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Therefore, the features defined with "first", "second" can explicitly or implicitly include at least one of the features. In the description of the present application, the meaning of "a plurality of" is at least two, for example, two, three, etc., unless otherwise specifically limited.

[0102] Any processes or methods described in the flowcharts or otherwise described herein can be understood as representing modules, segments, or portions of code that include one or more executable instructions for implementing specific logical functions or steps, and the various embodiments of the application can include additional or fewer steps performing the same or equivalent functions as those shown or discussed, in different orders, including substantially simultaneous execution of the functions described with respect to particular steps, and the like.

[0103] The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a list of executable instructions for implementing the logic function, and can be embodied in any computer-readable medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, processor- containing system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions, or a combination thereof. For the purposes of this specification, a "computer-readable medium" can be any apparatus that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device. The computer-readable medium can specifically include the following, which are non-exhaustive listings: electrical connections (electrical apparatus) having one or more wires, a portable computer diskette (magnetic apparatus), a random-access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, and a portable compact disc read-only memory (CDROM). Additionally, the computer-readable medium can even be paper or another suitable medium upon which the program is printed, since the program can be electronically captured, for example, by optically scanning the paper or other medium, then electronically converted into a form that is suitable for use in a computer storage medium.

[0104] It should be understood that portions of the application can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in a memory and executed by a suitable instruction execution system. As such, if implemented in hardware, and in another embodiment, any of the following technologies, known in the art, or a combination thereof, can be used: discrete logic circuitry having logic gates for implementing logic functions on data signals, application specific integrated circuits having appropriate combinational logic gates, programmable gate arrays (PGA), field programmable gate arrays (FPGA), and the like.

[0105] Those skilled in the art of the present technology can understand that all or part of the steps carried out by the above-mentioned embodiment method can be completed by programs instructing related hardware, and the programs can be stored in a computer readable storage medium. When the program is executed, it includes one of the steps of the method embodiment or a combination thereof.

[0106] In addition, each functional unit in each embodiment of the present application can be integrated into one processing module, or each unit can exist physically alone, or two or more units can be integrated into one module. The integrated module can be realized in the form of hardware or in the form of a software functional module. When the integrated module is realized in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer readable storage medium.

[0107] The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it should be understood that the above-mentioned embodiments are exemplary and cannot be understood as limiting the present application, and those skilled in the art can make changes, modifications, replacements and variations to the above-mentioned embodiments within the scope of the present application.

Claims

1. A method of estimating a three-dimensional pose of a finger from a planar fingerprint, the method comprising: The method comprises the following steps: Collecting a planar fingerprint image to be tested; Inputting the planar fingerprint image to be tested into a preset deep neural network model to output a predicted attitude angle; wherein the planar fingerprint image to be tested is subjected to feature extraction through stacking and abstraction of a convolution layer and a nonlinear activation layer to output a fixed-dimension feature descriptor; an attention mechanism module is used to perform attention mechanism calculation on the fixed-dimension feature descriptor, fuse an output heat map with the feature descriptor, and obtain an optimized feature descriptor; a value range of a pitch angle, a roll angle and a yaw angle of the optimized feature descriptor is divided into a plurality of intervals, and then a three-dimensional angle prediction module is used to output a probability that a current fingerprint image belongs to each interval, and a final predicted attitude angle is obtained based on the probability and a midpoint value of the divided interval and weighted; The deep neural network model comprises a feature extraction backbone network, an attention mechanism module and a three-dimensional angle prediction module, and the feature extraction backbone network comprises a convolution layer and a nonlinear activation layer; Obtaining training data, including a database of planar fingerprint images and corresponding finger three-dimensional attitudes; Using the training data, a loss function is used to train the deep neural network module; The loss function is represented as: L roll / pitch / yaw = L ce + λL MSE L all = L roll + L pitch + L yaw where L CE is the cross-entropy loss function for classification problems, y c is either 0 or 1, y c = 1 if and only if c is the current pose angle distribution in the c-th interval, p c c∈[1,M] represents the probability value predicted by the network belonging to each interval, s c represents the representative value of each angle interval, and M represents the total number of intervals, is the angle prediction value, θ is the angle true value, L roll / pitch / yaw is the loss function for a single pose angle, λ is a hyperparameter, and is taken as 1.0, L all is the total loss function for all pose angles.

2. The method of claim 1, wherein, The obtaining of the training data comprises: Rolling a finger bound with a gyroscope on a fingerprint sensor from one side to the other side; A fingerprint collector is controlled by a program to collect a plurality of planar fingerprint images at a fixed frequency, and the gyroscope synchronously reads out the angle; The three-dimensional attitude angle corresponding to each planar fingerprint image is calculated based on the difference between the values of the two gyroscopes.

3. The method of claim 1, wherein, The obtaining of the training data comprises using an optical tracking technology in combination with a fingerprint collector to synchronously obtain a finger attitude and a fingerprint image, or using a three-dimensional fingerprint database constructed in advance to synthesize a planar fingerprint image under any attitude.

4. An apparatus for estimating a three-dimensional pose of a finger from a planar fingerprint, the apparatus comprising: The method comprises a data collection module and a prediction module, wherein The data collection module is configured to collect a planar fingerprint image to be tested; The prediction module is configured to predict the planar fingerprint image to be tested to output a predicted attitude angle; wherein the planar fingerprint image to be tested is subjected to feature extraction through stacking and abstraction of a convolution layer and a nonlinear activation layer to output a fixed-dimension feature descriptor; an attention mechanism module is used to perform attention mechanism calculation on the fixed-dimension feature descriptor, fuse an output heat map with the feature descriptor, and obtain an optimized feature descriptor; a value range of a pitch angle, a roll angle and a yaw angle of the optimized feature descriptor is divided into a plurality of intervals, and then a three-dimensional angle prediction module is used to output a probability that a current fingerprint image belongs to each interval, and a final predicted attitude angle is obtained based on the probability and a midpoint value of the divided interval and weighted; The deep neural network model comprises a feature extraction backbone network, an attention mechanism module and a three-dimensional angle prediction module, and the feature extraction backbone network comprises a convolution layer and a nonlinear activation layer; Obtaining training data, including a database of planar fingerprint images and corresponding finger three-dimensional attitudes; The deep neural network module is trained using the training data and a loss function; The loss function is expressed as: L roll / pitch / yaw = L ce + λL MSE L all = L roll + L pitch + L yaw where L CE is the cross-entropy loss function for classification problems, y c is either 0 or 1, y c = 1 if and only if c is the current pose angle distribution is in the c-th interval, p c c∈[1,M] represents the probability value predicted by the network belonging to each interval, s c represents the representative value of each angle interval, and M represents the total number of intervals, is the angle prediction value, θ is the angle true value, L roll / pitch / yaw is the loss function for a single pose angle, λ is a hyperparameter, and is taken as 1.0, L all is the total loss function for all pose angles.

5. The apparatus of claim 4, wherein, The prediction module further comprises a training unit, which is specifically configured to: Obtain training data, including a database of planar fingerprint images and corresponding finger three-dimensional poses; The deep neural network module is trained using the training data and a loss function.

6. A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by a processor to implement the method of any one of claims 1-3.

Images