A contact type finger image and finger state true value synchronous acquisition method

Abstract

This application proposes a method for simultaneously acquiring contact-based finger images and true values ​​of finger states, relating to the fields of biometric recognition and human-computer interaction. The method includes: acquiring contact-based finger images using a first device equipped with a fingerprint or touch sensor, and simultaneously acquiring a sequence of non-contact images of the finger using a second device based on the principle of specular reflection or an additional device with imaging capabilities; using various visual algorithms to calculate true value data of the finger state based on the non-contact acquired finger image sequence; and optimizing the contact-based finger state measurement technique based on the true value data of the finger state. This application, while acquiring contact-based finger images using fingerprint or touch sensors in consumer electronic devices such as mobile phones, obtains true value data of the finger state by acquiring image sequences of the finger and adjacent hand regions at low cost and conveniently, thus enabling convenient acquisition of finger state data associated with contact-based finger images.

Description

Technical Field

[0001] This application relates to the fields of biometric recognition and human-computer interaction technology, and in particular to a method for synchronously acquiring contact finger images and finger state ground truth values. Background Technology

[0002] Currently, fingerprint recognition technology, as a mature biometric identification technology, has been widely used in the field of identity verification. However, the information contained in fingerprint images includes not only individual identity information, but also information such as the three-dimensional posture of the finger and the force applied by the finger. This information extracted from fingerprint images can be applied to the field of human-computer interaction to develop diverse interaction methods; it can also enable richer operations in identity authentication applications. Touch interaction technology has similar characteristics. In addition to the widely extracted and used contact and positioning information, touch images also contain various information such as the three-dimensional posture of the finger and the force applied by the finger, which can be used to develop more practical applications.

[0003] Many finger state measurement technologies based on contact sensors such as fingerprints or touch sensors employ deep network-based methods, thus relying on the collection of large amounts of finger samples and ground-value data. However, current mainstream technologies for acquiring ground-value finger states, such as optical tracking, are highly complex, requiring expensive equipment and cumbersome procedures. This makes it difficult to use these technologies to acquire ground-value data of the current user's finger state, and therefore difficult to use for personalized fine-tuning of pre-trained deep networks. In practical applications, how to conveniently and simultaneously acquire both contact finger images and ground-value data of finger states using mobile phones with fingerprint or touch sensors, thereby providing rich data for the design and optimization of algorithms used in the application, has become a worthy research problem. Summary of the Invention

[0004] This application aims to at least partially address one of the technical problems in the related art.

[0005] Therefore, the purpose of this application is to propose a method, device, electronic device and readable medium for synchronous acquisition of contact finger images and finger state truth values, which can simultaneously acquire non-contact image sequences of the finger while acquiring contact finger images using a mobile phone sensor, and obtain the truth data of the finger state accordingly, so as to optimize the finger state measurement technology based on contact images.

[0006] To achieve the above objectives, the first aspect of this application proposes a method for synchronously acquiring contact-type finger images and finger state ground truth, comprising:

[0007] A first device with a fingerprint or touch sensor acquires contact images of the finger, and a second device with additional imaging capabilities acquires a sequence of non-contact images of the finger simultaneously, based on the principle of specular reflection or additionally.

[0008] Using a variety of visual algorithms, ground truth data of finger states are calculated based on the sequence of finger images acquired non-contactly.

[0009] Based on the true value data of the finger state, the contact-based finger state measurement technology is optimized.

[0010] Optionally, the step of acquiring contact-type finger images through a first device with a fingerprint or touch sensor, and simultaneously acquiring a sequence of non-contact finger images based on the principle of specular reflection, includes:

[0011] Place the first device face up on a table or fix its position in another way;

[0012] The reflector is fixed to the upper area of ​​the front of the first device, with the mirror surface of the reflector facing the screen of the first device;

[0013] When a user places their finger on the fingerprint or touch sensor of the first device to capture a contact-based finger image, the front-facing camera of the first device simultaneously captures a sequence of non-contact images of the finger through the reflection of a mirror.

[0014] Optionally, the step of acquiring contact-based finger images via a device with a fingerprint or touch sensor, and simultaneously acquiring a sequence of non-contact finger images via an additional second device with imaging capabilities, includes:

[0015] Place the first device face up on a table or fix its position in another way;

[0016] Secure the second device to the upper area of ​​the front of the first device;

[0017] When a user places their finger on the fingerprint or touch sensor of the first device to acquire a contact-based finger image, the second device simultaneously acquires a sequence of non-contact images of the finger.

[0018] Optional, also includes:

[0019] When acquiring non-contact image sequences of fingers, supplementary light sources are used for auxiliary illumination, depending on the actual situation.

[0020] Optionally, the ground truth data of the finger state includes the finger position, three-dimensional posture, force, and other data related to the information contained in the contact finger image, wherein the three-dimensional posture of the finger refers to the three-dimensional posture of the first segment of the finger.

[0021] Optionally, the step of using multiple visual algorithms to calculate ground truth data of the finger state based on the non-contactly acquired finger image sequence includes:

[0022] Using deep learning methods, a visual recognition model is built to identify the finger positions;

[0023] Based on the spatial relationship and camera parameters, the three-dimensional pose of the finger is calculated.

[0024] Using deep learning methods, the force of the finger is estimated based on the changes in the shape of the fingernail.

[0025] Optionally, the contact-based finger state measurement technology includes:

[0026] The three-dimensional pose, force, and finger position of the finger are measured from contact finger images using deep learning methods.

[0027] To achieve the above objectives, a second aspect of this application provides a device for synchronously acquiring contact-type finger images and finger state truth values, comprising:

[0028] The acquisition module is used to acquire contact finger images through a first device with a fingerprint or touch sensor, and simultaneously acquire non-contact image sequences of the finger based on the principle of specular reflection or an additional second device with a shooting function;

[0029] The calculation module is used to calculate the true value data of the finger state based on the non-contact acquired finger image sequence using various vision algorithms;

[0030] The optimization module is used to optimize the contact finger state measurement technology based on the true value data of the finger state.

[0031] To achieve the above objectives, a third aspect of this application provides an electronic device, including: a processor, and a memory communicatively connected to the processor;

[0032] The memory stores computer-executed instructions;

[0033] The processor executes computer execution instructions stored in the memory to implement the method as described in any one of the first aspects above.

[0034] To achieve the above objectives, a fourth aspect of this application provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, are used to implement the method as described in any one of the first aspects above.

[0035] The technical solutions provided by the embodiments of this application bring at least the following beneficial effects:

[0036] By proposing two types of acquisition schemes, namely, using the principle of specular reflection or additional devices with shooting functions, it is possible to simultaneously acquire non-contact image sequences of the finger (which may include adjacent hand areas) while the user is acquiring contact finger images using contact sensors (including fingerprint sensors and touch sensors) of consumer electronic devices such as mobile phones, tablets, computers, and smartwatches. Through visual algorithms, various ground truth data of the finger state can be calculated from the non-contact images, and the ground truth data can be used to optimize the contact finger state measurement technology.

[0037] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0038] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0039] Figure 1 This is a flowchart illustrating a method for synchronously acquiring contact finger images and finger state truth values ​​according to an embodiment of this application;

[0040] Figure 2 This is a schematic diagram illustrating the use of specular reflection to collect finger data according to an embodiment of this application;

[0041] Figure 3 This is a schematic diagram illustrating the use of additional imaging equipment to collect finger data according to an embodiment of this application;

[0042] Figure 4 This is a flowchart illustrating the use of specular reflection to collect finger data according to an embodiment of this application;

[0043] Figure 5 This is a flowchart illustrating the use of additional imaging equipment to collect finger data according to an embodiment of this application;

[0044] Figure 6 This is a block diagram illustrating a synchronous acquisition device for contact finger images and finger state truth values ​​according to an embodiment of this application. Detailed Implementation

[0045] The embodiments of this application are described in detail below. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0046] The following description, with reference to the accompanying drawings, describes a method and apparatus for synchronously acquiring contact finger images and finger state truth values ​​according to embodiments of this application.

[0047] Figure 1 This is a flowchart illustrating a method for synchronously acquiring contact finger images and finger state truth values ​​according to an embodiment of this application, as shown below. Figure 1 As shown, the method includes the following steps:

[0048] Step 101: Acquire contact finger images using a first device with a fingerprint or touch sensor, and simultaneously acquire non-contact finger image sequences using a second device based on the principle of specular reflection or an additional device with a shooting function.

[0049] Existing technologies suffer from the following technical problems: Current mainstream technologies for acquiring true finger state values, such as optical tracking, are very complex, requiring expensive equipment and cumbersome operation processes. These technologies are difficult to use for acquiring true finger state data of the current user, and therefore difficult to use for personalized fine-tuning of pre-trained deep networks.

[0050] Therefore, to address the above issues, this application proposes two types of acquisition schemes: acquiring non-contact image sequences of the fingers based on the principle of specular reflection or additional devices with imaging capabilities, and then calculating the true value data of the finger state using various visual algorithms.

[0051] In this application embodiment, the first device can be a consumer electronic device such as a mobile phone, tablet computer, computer, or smartwatch that includes a fingerprint sensor or touch sensor, and the second device can be a mobile phone, camera, smart glasses, or other wearable device with a shooting function. This application does not limit the device in this regard.

[0052] In one possible embodiment, a schematic diagram of using specular reflection to collect finger data is shown below. Figure 2 As shown in the diagram, a schematic diagram of collecting finger data using additional imaging equipment is presented. Figure 3 In this embodiment, the first device is a mobile phone equipped with a fingerprint sensor or a touch sensor, and the second device is a mobile phone with a camera function or other camera device.

[0053] Figure 4 This is a flowchart illustrating the use of specular reflection to collect finger data according to an embodiment of this application, including:

[0054] Step 201: Place the first device face up on the table or fix its position in another way.

[0055] If the first device is a mobile phone equipped with a fingerprint sensor or touch sensor, such as Figure 2 As shown, place the phone face up on a table, or fix its position in another way.

[0056] It is understood that there are various methods to fix the location of a mobile phone, and this application does not limit this method.

[0057] Step 202: Fix the reflector to the upper area of ​​the front of the first device, with the mirror surface of the reflector facing the screen of the first device.

[0058] like Figure 2 As shown, the mirror is fixed to the upper area of ​​the front of the phone, with its mirror surface facing the phone screen.

[0059] It is understood that there are various methods for fixing the position of a reflector, and this application does not limit the method.

[0060] In one possible embodiment, a special bracket can be used to fix the mirror to the upper area of ​​the front of the phone.

[0061] Step 203: When the user places their finger on the fingerprint or touch sensor of the first device to collect a contact-type finger image, the front-facing camera of the first device is used to simultaneously collect a non-contact image sequence of the finger through the reflection of the mirror.

[0062] like Figure 2 As shown, when a user places their finger on the fingerprint or touch sensor of a mobile phone to collect a contact-based finger image, the front-facing camera of the same mobile phone can be used to simultaneously collect a sequence of non-contact images of the finger through the reflection of the mirror.

[0063] That is, while using the phone's sensors to capture contact images of the finger, the front-facing camera of the same phone is used to simultaneously capture a sequence of non-contact images of the finger.

[0064] Figure 5 This is a flowchart illustrating the use of additional imaging equipment to collect finger data according to embodiments of this application, including:

[0065] Step 301: Place the first device face up on the table, or fix its position in another way.

[0066] Similar to the above process, place the phone face up on a table or fix its position in another way.

[0067] It is understood that there are various methods to fix the location of a mobile phone, and this application does not limit this method.

[0068] Step 302: Fix the second device in the upper area of ​​the front of the first device.

[0069] It is understood that there are various methods for fixing the position of a device with a shooting function, and this application does not limit this method.

[0070] In one possible embodiment, a special bracket is used to fix another mobile phone or other camera device with a camera function to the upper area of ​​the front of the mobile phone.

[0071] Step 303: When the user places their finger on the fingerprint or touch sensor of the first device to acquire a contact-type finger image, the second device simultaneously acquires a non-contact image sequence of the finger.

[0072] like Figure 3 As shown, when a user places their finger on the fingerprint or touch sensor of a mobile phone to capture a contact-based finger image, the front-facing camera of another mobile phone or other shooting device can be used to simultaneously capture a non-contact image sequence of the finger.

[0073] That is, while using the phone's sensors to capture contact images of the finger, the front-facing camera of another phone or other shooting device simultaneously captures a sequence of non-contact images of the finger.

[0074] In addition, for Figure 4 and Figure 5 In order to achieve better image capture quality, the method shown can also use supplementary light sources for auxiliary illumination when acquiring non-contact image sequences of fingers, depending on the actual situation.

[0075] Step 102: Using multiple visual algorithms, calculate the ground truth data of the finger state based on the finger image sequence acquired non-contactly.

[0076] In this embodiment of the application, the true data of the finger state includes the finger position (e.g., the right index finger), three-dimensional pose, force (normal force and shear force), and other data related to the information contained in the contact finger image, wherein the three-dimensional pose of the finger refers to the three-dimensional pose of the first finger segment.

[0077] It is understandable that, using a reasonable visual algorithm, ground truth data (or data very close to the truth) of finger states can be obtained from a sequence of finger images. This application does not impose specific restrictions on the visual algorithm used; any visual algorithm capable of calculating and obtaining ground truth data of finger states may be employed.

[0078] The following are some algorithmic solutions that can be referenced, but are not intended to limit this application:

[0079] (1) Use deep learning methods to build a visual recognition model to identify the finger position. Alternatively, when collecting fingerprints, distinguish the fingerprints by specifying the finger position to be collected.

[0080] As one possible implementation, training a deep learning model can automatically identify the position and identity of a finger (such as the right index finger). This can be achieved using a convolutional neural network (CNN) or other deep learning architectures.

[0081] Alternatively, fingerprints can be differentiated by specifying the finger position during fingerprint collection. For example, users can pre-specify which finger to place on the sensor, thus enabling differentiation.

[0082] (2) Based on the spatial position relationship and camera parameters, the three-dimensional pose of the finger is obtained by calculation.

[0083] One possible approach is to utilize spatial geometric relationships and camera parameters (such as focal length and pixel size) to calculate the finger's pose in three-dimensional space through multi-view geometric reconstruction techniques. This can be achieved using structured light, stereo vision, or multi-view geometric methods.

[0084] (3) Using deep learning methods, the force of the finger is estimated based on the changes in the shape of the fingernail.

[0085] One possible approach is to analyze the morphological changes of the fingernail using a deep learning model to estimate the forces acting on the finger (including normal and shear forces). This method can be based on convolutional neural networks or other deep learning models, trained to recognize changes in force.

[0086] In this embodiment, interpolation and other methods can be used to further achieve more accurate data synchronization for the collected contact finger images and finger state ground truth data. A reasonable data storage and representation format is designed according to actual needs to store the contact finger images and finger state ground truth data, thereby facilitating subsequent data use.

[0087] In addition, to improve the accuracy and convenience of data acquisition, an intuitive user interface and interactive flow can be designed to guide users in correctly placing their fingers and performing other operations. The acquisition system can be designed to calculate the true value of the finger's state in real time, thereby providing real-time feedback, such as acquisition status prompts and image acquisition quality prompts, to help users understand the current operation status.

[0088] Step 103: Optimize the contact-based finger state measurement technology based on the true value data of the finger state.

[0089] Finally, the true value data of finger states obtained through the aforementioned steps are used to optimize the contact-based finger state measurement technology.

[0090] It should be noted that the contact-based finger state measurement technology involved here includes, but is not limited to: measuring the three-dimensional pose, force, and finger position of the finger from fingerprint or touch images using deep learning methods.

[0091] To achieve the above embodiments, this application also proposes a device for synchronously acquiring contact finger images and finger state truth values.

[0092] Figure 6 This is a block diagram of a contact-type finger image and finger state truth synchronous acquisition device 10 according to an embodiment of this application, comprising:

[0093] The acquisition module 100 is used to acquire contact finger images through a first device with a fingerprint or touch sensor, and simultaneously acquire non-contact image sequences of the finger based on the principle of specular reflection or an additional second device with a shooting function;

[0094] The calculation module 200 is used to calculate the true value data of the finger state based on the finger image sequence acquired non-contactly using a variety of vision algorithms.

[0095] The optimization module 300 is used to optimize the contact finger state measurement technology based on the true value data of the finger state.

[0096] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0097] To implement the above embodiments, this application also proposes an electronic device, including: a processor and a memory communicatively connected to the processor; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory to implement the method provided in the foregoing embodiments.

[0098] To implement the above embodiments, this application also proposes a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the methods provided in the foregoing embodiments.

[0099] In the foregoing descriptions of the embodiments, the terms "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0100] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0101] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0102] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0103] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0104] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0105] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0106] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of this application.

[0107] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this application can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this application can be achieved, and this is not limited herein.

[0108] The specific embodiments described above do not constitute a limitation on the scope of protection of this application. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A method for synchronously acquiring contact-based finger images and finger state ground truth values, characterized in that, include: A first device with a fingerprint or touch sensor acquires contact images of the finger, and a second device with additional imaging capabilities acquires a sequence of non-contact images of the finger simultaneously, based on the principle of specular reflection or additionally. Using a variety of visual algorithms, ground truth data of finger states are calculated based on the sequence of finger images acquired non-contactly. Based on the true value data of the finger state, the contact-based finger state measurement technology is optimized; The process of acquiring contact-type finger images through a first device equipped with a fingerprint or touch sensor, and simultaneously acquiring a sequence of non-contact finger images based on the principle of specular reflection, includes: Place the first device face up on a table or fix its position in another way; The reflector is fixed to the upper area of ​​the front of the first device, with the mirror surface of the reflector facing the screen of the first device; When a user places their finger on the fingerprint or touch sensor of the first device to capture a contact-based finger image, the front-facing camera of the first device simultaneously captures a sequence of non-contact images of the finger through the reflection of a mirror. The process of acquiring contact-based finger images via a device with a fingerprint or touch sensor, and simultaneously acquiring a sequence of non-contact finger images via a second device with additional imaging capabilities, includes: Place the first device face up on a table or fix its position in another way; Secure the second device to the upper area of ​​the front of the first device; When a user places their finger on the fingerprint or touch sensor of the first device to acquire a contact-based finger image, the second device simultaneously acquires a sequence of non-contact images of the finger. The true data of the finger state includes the finger position, three-dimensional posture, force, and other data related to the information contained in the contact finger image, wherein the three-dimensional posture of the finger refers to the three-dimensional posture of the first finger segment. The method employs multiple visual algorithms to calculate ground truth data of the finger state based on the non-contactly acquired finger image sequence, including: Using deep learning methods, a visual recognition model is built to identify the finger positions; Based on the spatial relationship and camera parameters, the three-dimensional pose of the finger is calculated. Using deep learning methods, the force of the finger is estimated based on the shape changes of the fingernail. The contact-based finger state measurement technology includes: The three-dimensional pose, force, and finger position of the finger are measured from contact finger images using deep learning methods.

2. The method according to claim 1, characterized in that, Also includes: When acquiring non-contact image sequences of fingers, supplementary light sources are used for auxiliary illumination, depending on the actual situation.

3. A device for synchronously acquiring contact finger images and finger state truth values ​​based on the method of any one of claims 1-2, characterized in that, include: The acquisition module is used to acquire contact finger images through a first device with a fingerprint or touch sensor, and simultaneously acquire non-contact image sequences of the finger based on the principle of specular reflection or an additional second device with a shooting function; The calculation module is used to calculate the true value data of the finger state based on the non-contact acquired finger image sequence using various vision algorithms; The optimization module is used to optimize the contact finger state measurement technology based on the true value data of the finger state.

4. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1-2.

5. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-2.

Images