A symbol input method based on finger print recognition

Abstract

The application provides a symbol input method based on finger fingerprint recognition, and relates to the technical field of symbol input. The method is applied to a symbol input system, and the symbol input system comprises a fingerprint image acquisition module, a finger positioning module, a symbol mapping module and a feedback module. The method comprises the following steps: collecting, by the fingerprint image acquisition module, a fingerprint image of a user's finger that strikes a touch screen; comparing, by the finger positioning module, the collected fingerprint image with a stored fingerprint template, and identifying finger information and striking position information of the fingerprint image; mapping, by the symbol mapping module, the finger information and the striking position information into a symbol according to a predefined symbol mapping relationship; and forming, by the feedback module, visual feedback based on the symbol, and displaying the symbol on a display screen.

Description

Technical Field

[0001] This application relates to the field of symbol input technology, and in particular to a symbol input method based on fingerprint recognition. Background Technology

[0002] Currently, touchscreens remain the primary input method for mobile devices such as smartphones and tablets. However, staring at the on-screen keyboard for extended periods to input symbols can easily lead to eye strain. Furthermore, for visually impaired users, touchscreen input, which relies too heavily on visual feedback, is not user-friendly.

[0003] Furthermore, the input efficiency of touchscreen devices is often limited by the user's proficiency. Beginners may spend a significant amount of time recognizing the letter layout and locating specific keys. Touchscreen input methods can only achieve high efficiency when users can accurately see the position of their fingers and the specific keys on the phone's keyboard. While voice and gesture-based input methods exist, each has its own limitations in its application environment. Therefore, to improve input efficiency and user experience, new input methods that do not rely on visual key positioning need to be developed. Summary of the Invention

[0004] In view of the above problems, embodiments of this application provide a symbol input method and system based on fingerprint recognition, so as to overcome the above problems or at least partially solve the above problems.

[0005] A first aspect of this application provides a symbol input method based on fingerprint recognition, applied to a symbol input system. The symbol input system includes: a fingerprint image acquisition module, a finger positioning module, a symbol mapping module, and a feedback module. The method includes:

[0006] The fingerprint image acquisition module acquires fingerprint images of the user's finger tapping on the touchscreen.

[0007] Using the finger positioning module, the acquired fingerprint image is compared with the stored fingerprint template to identify the finger information and tapping position information of the fingerprint image;

[0008] The symbol mapping module maps the finger information and the tapping position information into symbols according to a predefined symbol mapping relationship.

[0009] The feedback module generates visual feedback based on the symbol and displays the symbol on the screen.

[0010] In one possible implementation, the step of using the finger positioning module to compare the acquired fingerprint image with a stored fingerprint template to identify the finger information of the fingerprint image includes:

[0011] The fingerprint image is matched one by one with the stored fingerprint templates using the finger positioning module to determine the finger type of the user's finger in the fingerprint image; the finger type includes at least: left hand, right hand, index finger, ring finger, little finger, thumb, and middle finger;

[0012] The finger part to which the fingerprint image belongs is determined, and the finger part includes at least: fingertip, finger pad, left finger side, and right finger side.

[0013] In one possible implementation, the symbol input system further includes: a fingerprint registration module; the method further includes:

[0014] For first-time registered users, the fingerprint registration module activates the fingerprint sensor to acquire registration fingerprint images of different fingers of the registered user; the registration fingerprint image is a complete rolling fingerprint image for a single finger, and / or a planar fingerprint image for each part of the finger for a single finger;

[0015] Fingerprint features are extracted from the registered fingerprint image to generate fingerprint templates for each finger of the registered user;

[0016] The fingerprint templates of all the registered user's fingers are stored in the fingerprint database.

[0017] In one possible implementation, after acquiring the registered fingerprint image, the method further includes:

[0018] The fingerprint quality of the registered fingerprint image is assessed to determine its quality.

[0019] If the fingerprint quality of the registered fingerprint image is lower than a preset quality threshold, the registered fingerprint image of that finger is re-acquired.

[0020] In one possible implementation, the tapping location information of the fingerprint image is obtained through the following steps:

[0021] When the fingerprint sensor is a large-area sensor that can accommodate multiple fingers, the touch screen of the fingerprint sensor is divided into multiple areas. The area information corresponding to the touch screen where the user's finger taps in the fingerprint image is obtained, and the area information is used as the tapping position information.

[0022] When the fingerprint sensor is a combination of multiple small-area sensors that can only accommodate a single finger, the identity information of the small-area sensor that captures the fingerprint image is obtained, and the identity information of the small-area sensor is used as the tapping position information.

[0023] In one possible implementation, the method further includes:

[0024] When the fingerprint sensor is a small-area sensor that only supports output from a single finger, symbol mapping is performed based on the finger type and finger location to generate the symbol mapping relationship;

[0025] When the fingerprint sensor is a large-area sensor that can accommodate multiple fingers, the touch screen of the fingerprint sensor is divided into multiple areas, and symbol mapping is performed by combining the finger type, finger position and different areas of the touch screen to generate the symbol mapping relationship;

[0026] When the fingerprint sensor is a combination of multiple small-area sensors that can only accommodate a single finger, symbol mapping is performed by combining the finger type, finger location, and different combinations of the small-area sensors to generate the symbol mapping relationship.

[0027] In one possible implementation, matching the fingerprint image with fingerprint templates stored in a fingerprint database includes:

[0028] The finger positioning module uses a feature extraction algorithm to extract features from the fingerprint image to obtain fingerprint features; the feature extraction algorithm is at least one of the following: minutiae feature extraction algorithm, local feature extraction algorithm, ridge feature extraction algorithm, and deep learning descriptor extraction algorithm.

[0029] The finger positioning module uses a matching algorithm to match the fingerprint features with fingerprint templates stored in the fingerprint database. The matching algorithm is at least one of the following: sliding window search method, correlation matching method, feature point matching method, and similarity distance judgment method.

[0030] In one possible implementation, the method further includes:

[0031] Through the feedback module, sound feedback is generated based on the symbols, and / or,

[0032] Vibration feedback is generated based on the symbol through the feedback module.

[0033] In one possible implementation, the method further includes:

[0034] When the fingerprint image includes multiple fingerprints, the fingerprint image is split into multiple independent fingerprint images. Using the finger positioning module, each independent fingerprint image is matched with a fingerprint template stored in the fingerprint database to obtain the finger information and tapping position information in each independent fingerprint image.

[0035] In one possible implementation, after identifying the finger information and the tapping location information, the method further includes:

[0036] The tapping area is determined based on the finger information and the tapping location information;

[0037] The striking area is matched with a predefined symbol mapping table to obtain the symbol corresponding to the striking area;

[0038] The symbol corresponding to the tapping area is output to the display device.

[0039] A second aspect of this application also proposes a symbol input system based on fingerprint recognition, the system comprising:

[0040] The fingerprint image acquisition module is used to acquire fingerprint images of the user's finger tapping on the touch screen;

[0041] The finger positioning module is used to compare the acquired fingerprint image with the stored fingerprint template to identify the finger information and tapping position information of the fingerprint image;

[0042] The symbol mapping module is used to map the finger information and the tapping position information into symbols according to a predefined symbol mapping relationship;

[0043] A feedback module is used to generate visual feedback based on the symbol and display the symbol on a display screen.

[0044] In one possible implementation, the finger positioning module includes:

[0045] The finger type determination submodule is used to match the fingerprint image with the stored fingerprint templates one by one to determine the finger type of the user's finger in the fingerprint image; the finger type includes at least: left hand, right hand, index finger, ring finger, little finger, thumb, and middle finger;

[0046] The finger location determination submodule is used to determine the finger location to which the fingerprint image belongs. The finger location includes at least: fingertip, finger pad, left finger side, and right finger side.

[0047] In one possible implementation, the system further includes: a fingerprint registration module, used for:

[0048] For first-time registered users, the fingerprint sensor is activated to acquire registered fingerprint images of different fingers of the registered user; the registered fingerprint image is a complete rolling fingerprint image for a single finger, and / or a planar fingerprint image for each part of the finger for a single finger;

[0049] Fingerprint features are extracted from the registered fingerprint image to generate fingerprint templates for each finger of the registered user;

[0050] The fingerprint templates of all the registered user's fingers are stored in the fingerprint database.

[0051] In one possible implementation, the system further includes:

[0052] The quality assessment module is used to assess the fingerprint quality of the registered fingerprint image after it has been acquired, and to determine the fingerprint quality of the registered fingerprint image.

[0053] The reminder module is used to re-acquire the registered fingerprint image of the finger if the fingerprint quality of the registered fingerprint image is lower than a preset quality threshold.

[0054] In one possible implementation, the finger positioning module includes:

[0055] The first tapping location information acquisition submodule is used to acquire the area information corresponding to the touch screen tapped by the user's finger in the fingerprint image when the fingerprint sensor is a large-area sensor that can accommodate multiple fingers, and the touch screen of the fingerprint sensor is divided into multiple areas, and to use the area information as the tapping location information.

[0056] The second tapping location information acquisition submodule is used to acquire the identity information of the small-area sensor that collects the fingerprint image when the fingerprint sensor is a combination of multiple small-area sensors that can only accommodate a single finger, and use the identity information of the small-area sensor as the tapping location information.

[0057] In one possible implementation, the system further includes:

[0058] The first symbol mapping relationship generation module is used to generate the symbol mapping relationship by performing symbol mapping according to the finger type and finger location when the fingerprint sensor is a small-area sensor that only supports output from a single finger.

[0059] The second symbol mapping relationship generation module is used to divide the touch screen of the fingerprint sensor into multiple areas when the fingerprint sensor is a large-area sensor that can accommodate multiple fingers, and perform symbol mapping based on finger type, finger position and different areas of the touch screen to generate the symbol mapping relationship.

[0060] The third symbol mapping relationship generation module is used to generate the symbol mapping relationship by combining different combinations of finger type, finger location and the small area sensors when the fingerprint sensor is a combination of multiple small area sensors that can only accommodate a single finger.

[0061] In one possible implementation, the finger positioning module is further configured to extract features from the fingerprint image using a feature extraction algorithm to obtain fingerprint features; the feature extraction algorithm is at least one of minutiae feature extraction algorithm, local feature extraction algorithm, ridge feature extraction algorithm, and deep learning descriptor extraction algorithm.

[0062] The finger positioning module is further configured to use a matching algorithm to match the fingerprint features with fingerprint templates stored in the fingerprint database. The matching algorithm is at least one of sliding window search, correlation matching, feature point matching, and similarity distance judgment.

[0063] In one possible implementation, the feedback module is further configured to generate sound feedback based on the symbol, and / or generate vibration feedback based on the symbol.

[0064] In one possible implementation, the finger positioning module is further configured to, when the fingerprint image includes multiple fingerprints, split the fingerprint image into multiple independent fingerprint images, match each independent fingerprint image with a fingerprint template stored in a fingerprint database, and obtain finger information and tapping position information in each independent fingerprint image.

[0065] In one possible implementation, the device further includes:

[0066] The tapping area determination module is used to determine the tapping area based on the finger information and the tapping position information after the finger information and tapping position information are identified.

[0067] The matching module is used to match the tapping area with a predefined symbol mapping table to obtain the symbol corresponding to the tapping area;

[0068] The display module is used to output the symbol corresponding to the tapping area to the display device.

[0069] A third aspect of this application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the symbol input method based on fingerprint recognition described in the first aspect of this application.

[0070] The fourth aspect of this application also provides a computer-readable storage medium having a computer program / instructions stored thereon, which, when executed by a processor, implements the steps of the symbol input method based on fingerprint recognition described in the first aspect of this application.

[0071] The fifth aspect of this application also provides a computer program product that, when run on an electronic device, causes a processor to execute the steps in the symbol input method based on fingerprint recognition as described in the first aspect of this application.

[0072] This application provides a symbol input method based on fingerprint recognition, applied to a symbol input system. The symbol input system includes a fingerprint image acquisition module, a finger positioning module, a symbol mapping module, and a feedback module. The method includes: acquiring a fingerprint image of a user's finger tapping on a touchscreen using the fingerprint image acquisition module; comparing the acquired fingerprint image with a stored fingerprint template using the finger positioning module to identify the finger information and tapping position information of the fingerprint image; mapping the finger information and tapping position information into a symbol using the symbol mapping module according to a predefined symbol mapping relationship; and generating visual feedback based on the symbol using the feedback module to display the symbol on the display screen.

[0073] This application acquires fingerprint images of users tapping on a touchscreen, matches these images with fingerprint templates stored in a fingerprint database, and thus identifies and locates the fingerprint, determining the finger information (e.g., which finger of the registered user the fingerprint belongs to) and the tapping location information indicated by the fingerprint image (the specific area on the touchscreen where the finger tapped). Therefore, based on a pre-defined symbol mapping relationship (the mapping relationship between finger information, tapping location information, and symbols), this application can determine the symbol indicated by the fingerprint image, display the symbol on the screen, and complete symbol input. By recognizing the tapping finger and tapping location, and utilizing a predefined symbol mapping relationship, this application enables finger typing input and command control without requiring the user to look at a keyboard. Simultaneously, by recognizing the fingerprint image, it assists in identity verification during symbol input, improving security. Attached Figure Description

[0074] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0075] Figure 1 This is a flowchart illustrating the steps of a symbol input method based on fingerprint recognition provided in an embodiment of this application.

[0076] Figure 2 This is a schematic diagram of the symbol mapping relationship of a two-finger sensor provided in an embodiment of this application;

[0077] Figure 3 This is a schematic flowchart of a fingerprint registration process provided in an embodiment of this application;

[0078] Figure 4 This is a schematic diagram of a symbol input process based on fingerprint recognition provided in an embodiment of this application;

[0079] Figure 5 This is a schematic diagram of the structure of a symbol input system provided in an embodiment of this application;

[0080] Figure 6 This is a schematic diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0081] Exemplary embodiments of this application will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of this application are shown in the drawings, it should be understood that this application may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of this application and to fully convey the scope of this application to those skilled in the art.

[0082] Currently, touchscreens remain the primary input method for mobile devices such as smartphones and tablets. However, prolonged staring at the on-screen keyboard for symbol input can easily lead to eye strain, and for visually impaired users, touchscreen input, which relies too heavily on visual feedback, is not user-friendly. Furthermore, the input efficiency of touchscreen devices is often limited by the user's proficiency; beginners may spend a significant amount of time recognizing letter layouts and locating specific keys. High input efficiency can only be achieved when users can accurately see the position of their fingers and the specific keys on the phone's keyboard. While voice and gesture-based input methods exist, each has its own limitations in its application environment.

[0083] In view of the above problems, this application proposes a symbol input method and system based on fingerprint recognition to solve the problems of over-reliance on visual feedback in touchscreen typing. The symbol input method based on fingerprint recognition provided in this application will be described in detail below with reference to the accompanying drawings and through some embodiments and application scenarios.

[0084] The first aspect of this application provides a symbol input method based on fingerprint recognition, applied to a symbol input system. The symbol input system includes: a fingerprint image acquisition module, a finger positioning module, a symbol mapping module, and a feedback module. (Refer to...) Figure 1 , Figure 1A flowchart illustrating the steps of a symbol input method based on fingerprint recognition provided in this application is shown below. Figure 1 As shown, the method includes:

[0085] Step S101: The fingerprint image of the user's finger tapping on the touch screen is acquired through the fingerprint image acquisition module.

[0086] In this embodiment, the symbol input system includes a fingerprint image acquisition module for acquiring fingerprint images. Specifically, the fingerprint image acquisition module includes a fingerprint sensor and a touchscreen adapted to the fingerprint sensor. The touchscreen of the fingerprint sensor can be placed in any suitable operating position on a corresponding electronic device such as a mobile phone, tablet, or watch.

[0087] When a user needs to input symbols, they tap the corresponding area on the touchscreen with their finger. The fingerprint sensor detects the tapping action, captures an image, and obtains the fingerprint image of the user's finger pressing on the touchscreen. In practical applications, when a user needs to perform a symbol input operation, a corresponding virtual keyboard can be displayed on the symbol input area of ​​the touchscreen, thereby assisting the user in accurately tapping the corresponding position on the touchscreen based on the virtual keyboard.

[0088] Furthermore, the fingerprint sensor in the fingerprint image acquisition module can be an optical sensor, an ultrasonic sensor, a capacitive sensor, etc. In this embodiment, the type of fingerprint sensor is not limited. The hardware carriers (such as fingerprint sensors, touchscreens, etc.) involved in this application embodiment are applicable to various electronic input devices, including but not limited to mobile phones, tablets, smartwatches, wearable devices, etc.

[0089] In one possible implementation, the method further includes:

[0090] When the fingerprint image includes multiple fingerprints, the fingerprint image is split into multiple independent fingerprint images. Using the finger positioning module, each independent fingerprint image is matched with a fingerprint template stored in the fingerprint database to obtain the finger information and tapping position information in each independent fingerprint image.

[0091] In practical applications, users can tap the touchscreen with a single finger, and the fingerprint image captured by the fingerprint image acquisition module will contain only the fingerprint of that single finger; users can also tap the touchscreen with multiple fingers at the same time, and the fingerprint image captured by the fingerprint image acquisition module will contain the fingerprints of multiple fingers simultaneously.

[0092] In this embodiment, when a fingerprint image contains multiple fingerprints, the fingerprint image is split into multiple independent fingerprint images. Specifically, the fingerprint image can be segmented into multiple smaller sub-images, such that each sub-image contains only the fingerprint of one finger. After obtaining multiple independent fingerprint images, step S102 is performed on each independent fingerprint image, using a finger positioning module to match each independent fingerprint image with fingerprint templates stored in the fingerprint database, thereby obtaining the finger information and tapping position information in each independent fingerprint image.

[0093] Step S102: Using the finger positioning module, the acquired fingerprint image is compared with the stored fingerprint template to identify the finger information and tapping position information of the fingerprint image.

[0094] After acquiring the fingerprint image, the fingerprint localization module performs fingerprint recognition and finger localization. Specifically, feature extraction is performed on the fingerprint image to obtain fingerprint features, which are then matched with fingerprint templates pre-stored in the fingerprint database. The fingerprint templates are fingerprints of the user's own finger that are stored in the fingerprint database during the user's registration process before first use.

[0095] In practical applications, there can be multiple registered users, resulting in multiple fingerprint templates belonging to different users stored in the fingerprint database. When matching fingerprint features with fingerprint templates one by one, the identity of the user currently performing the symbol input operation is determined based on the matching fingerprint template. If the fingerprint feature does not match any fingerprint templates in the fingerprint database, it is determined that the finger in the fingerprint image does not belong to a registered user, the symbol input operation is stopped, and an input failure reminder is generated.

[0096] In this embodiment, a finger positioning module matches the currently acquired fingerprint with the registered fingerprint to determine the finger's position. This finger position can be the left little finger, right index finger, or other finger types. Specifically, when the fingerprint features extracted from the fingerprint image successfully match the fingerprint templates stored in the fingerprint database, the finger information in the fingerprint image is determined based on the information corresponding to the fingerprint template. The finger information includes the type information of the currently pressing finger in the fingerprint image (e.g., the right index finger). The type information can be specifically divided into: left hand, right hand, index finger, ring finger, little finger, thumb, and middle finger. Correspondingly, each fingerprint template stored in the fingerprint database corresponds to a finger type of the registered user. During user registration, fingerprints need to be collected for each individual finger to generate an independent fingerprint template, thus ensuring that each fingerprint template corresponds to a finger type.

[0097] Furthermore, it is also necessary to obtain the tapping location information based on the fingerprint image. This tapping location information indicates the specific area on the touchscreen where the finger tapped; for example, the finger in the fingerprint image tapped the left side of the touchscreen. For instance, the specific tapping location information can be calculated based on the coordinates of the fingerprint in the fingerprint image.

[0098] In one possible implementation, matching the fingerprint image with fingerprint templates stored in a fingerprint database includes:

[0099] The finger positioning module uses a feature extraction algorithm to extract features from the fingerprint image to obtain fingerprint features; the feature extraction algorithm is at least one of the following: minutiae feature extraction algorithm, local feature extraction algorithm, ridge feature extraction algorithm, and deep learning descriptor extraction algorithm.

[0100] The finger positioning module uses a matching algorithm to match the fingerprint features with fingerprint templates stored in the fingerprint database. The matching algorithm is at least one of the following: sliding window search method, correlation matching method, feature point matching method, and similarity distance judgment method.

[0101] Optionally, the symbol input system also includes an auxiliary module for combining other inputs with biometric recognition technology to improve system security.

[0102] In one possible implementation, the step of using the finger positioning module to compare the acquired fingerprint image with a stored fingerprint template to identify the finger information of the fingerprint image includes:

[0103] The fingerprint image is matched one by one with the stored fingerprint templates using the finger positioning module to determine the finger type of the user's finger in the fingerprint image; the finger type includes at least: left hand, right hand, index finger, ring finger, little finger, thumb, and middle finger;

[0104] The finger part to which the fingerprint image belongs is determined, and the finger part includes at least: fingertip, finger pad, left finger side, and right finger side.

[0105] In this embodiment, the finger information includes: the type of the finger currently pressing in the fingerprint image (e.g., the finger is the right index finger), and the finger part information (e.g., the fingerprint belongs to the fingertip of the right index finger). Specifically, after matching the fingerprint image with fingerprint templates stored in the fingerprint database using the finger positioning module, the finger information of the fingerprint image is identified based on the matched fingerprint template. Specifically, after identifying and determining the finger type in the currently acquired fingerprint image (e.g., the finger performing the tapping operation in the fingerprint image is the right index finger), the finger part to which the fingerprint belongs can be further determined (e.g., the finger performing the tapping operation in the fingerprint image is the fingertip of the left middle finger or the pad of the right thumb). During actual tapping of the touchscreen, the user's finger may not be able to completely release the touchscreen, resulting in an incomplete fingerprint in the fingerprint image acquired by the fingerprint sensor. If the fingerprint database stores planar fingerprints or complete rolling fingerprints of the same finger in different postures, the finger part that touched the touchscreen can also be determined (e.g., fingertip, finger pad, left side, right side). Correspondingly, each fingerprint template stored in the fingerprint database corresponds to a finger part of a registered user's finger type. When a user registers their fingerprint, fingerprints need to be collected for each individual finger, resulting in planar fingerprints or complete rolling fingerprints of the same finger in different postures. For different parts of the finger (finger tip, finger pad, left side, right side), independent fingerprint templates are generated, so that each fingerprint template corresponds to a finger part of a finger type.

[0106] In one possible implementation, the tapping location information of the fingerprint image is obtained through the following steps:

[0107] When the fingerprint sensor is a large-area sensor that can accommodate multiple fingers, the touch screen of the fingerprint sensor is divided into multiple areas. The area information corresponding to the touch screen where the user's finger taps in the fingerprint image is obtained, and the area information is used as the tapping position information.

[0108] When the fingerprint sensor is a combination of multiple small-area sensors that can only accommodate a single finger, the identity information of the small-area sensor that captures the fingerprint image is obtained, and the identity information of the small-area sensor is used as the tapping position information.

[0109] In this embodiment, when acquiring the tapping position information of a finger in a fingerprint image, adjustments need to be made according to the size of the touchscreen in the fingerprint sensor and the requirements of symbol mapping. Specifically, for small-area sensors that can only accommodate one finger (i.e., the area of ​​the touchscreen adapted to the fingerprint sensor is smaller than the area of ​​two fingers), it is not suitable to further divide the touchscreen into regions, and the corresponding tapping position information is the same.

[0110] For large-area sensors that can accommodate multiple fingers simultaneously (i.e., the touchscreen area adapted to the fingerprint sensor is no less than the area of ​​two fingers), the touchscreen can be divided into several regions. Correspondingly, the tapping position information is used to indicate the specific area where the finger tapped in the fingerprint image. For example, if the fingerprint sensor's touchscreen can accommodate two fingers simultaneously, the touchscreen can be divided into left and right regions. The tapping position information obtained from the fingerprint image indicates whether the finger tapped the left or right region of the touchscreen.

[0111] In addition, a fingerprint sensor can be a combination of multiple small-area sensors that can only accommodate a single finger. The corresponding tapping position information is used to indicate which small-area sensor in the combination the finger tapped, that is, the identification information of the small-area sensor, such as the sensor number.

[0112] Step S103: The symbol mapping module maps the finger information and the tapping position information into symbols according to a predefined symbol mapping relationship.

[0113] In this embodiment, the symbol mapping module maps the finger information (one or more identified tapping finger positions) and tapping location information (tapping areas of each finger) of the acquired fingerprint image to corresponding symbols using a preset symbol mapping relationship. These symbols can be letter symbols, number symbols, punctuation marks, function symbols, etc.

[0114] In one possible implementation, the method further includes:

[0115] When the fingerprint sensor is a small-area sensor that only supports output from a single finger, symbol mapping is performed based on the finger type and finger location to generate the symbol mapping relationship;

[0116] When the fingerprint sensor is a large-area sensor that can accommodate multiple fingers, the touch screen of the fingerprint sensor is divided into multiple areas, and symbol mapping is performed by combining the finger type, finger position and different areas of the touch screen to generate the symbol mapping relationship;

[0117] When the fingerprint sensor is a combination of multiple small-area sensors that can only accommodate a single finger, symbol mapping is performed by combining the finger type, finger location, and different combinations of the small-area sensors to generate the symbol mapping relationship.

[0118] In this embodiment, before performing symbol input, it is necessary to define a symbol mapping relationship, that is, the mapping relationship between finger information, tapping position information, and symbols. During the user's symbol input process, the symbol mapping relationship can be displayed synchronously on the display screen to assist the user in performing the tapping action and further improve the accuracy of symbol input. Specifically, the symbol mapping method is related to the size of the touch screen in the fingerprint sensor. If the fingerprint sensor is a small-area sensor, that is, the area of ​​the touch screen adapted to the sensor is very small and only supports one finger input, the absolute position of the finger in the sensor area can be disregarded, and symbol mapping can be generated only based on the finger type (such as middle finger, little finger, thumb) and the part of the finger that touches the touch screen (fingertip, finger pad).

[0119] If the touchscreen of the fingerprint sensor is large enough to accommodate two fingers simultaneously, the touchscreen can be divided into two areas (left and right areas), and symbol mapping can be performed based on finger type and finger position. If the touchscreen is even larger, allowing more fingers to tap simultaneously, more areas can be divided (e.g., first area, second area, etc.), offering more choices of finger and area combinations. This allows for the selection of symbol mapping tables that are more ergonomic or even meet individual needs.

[0120] If the fingerprint sensor is a combination of multiple small-area sensors that can accommodate only a single finger, each small sensor can be considered a separate region. This allows for symbol mapping by combining region information, finger type, and finger location. Furthermore, users can define the mapping relationship between finger combinations, tapping positions, and symbols according to their specific needs.

[0121] In one possible implementation, taking a fingerprint sensor capable of accommodating two fingers simultaneously as an example, the touchscreen is divided into two areas (left and right areas), and symbol mapping is performed based on finger type and finger location. (See reference...) Figure 2 , Figure 2 A schematic diagram illustrating the symbol mapping relationship of a two-finger sensor is shown, such as... Figure 2 As shown, assuming the user holds the phone with their left hand, they mainly use their left thumb and the five fingers of their right hand for typing. Before first use, the user needs to register their fingerprints for their left thumb and the five fingers of their right hand, recording the fingerprint of each finger in the fingerprint database. In this embodiment, symbol mapping is performed by combining three parts of information from the fingerprint sensor: the touch screen area, the finger type, and the finger location. Specifically, in the touch screen area, left and right area information can be obtained; in the finger type part, six types of information are obtained: right thumb (0), right index finger (1), right middle finger (2), right ring finger (3), right little finger (4), and left thumb (5); in the finger location part, two types of information are obtained: fingertip (○) and fingertip (⌒).

[0122] For example, using the aforementioned fingerprint sensor, fingerprint images are captured when a user taps the touchscreen. The captured fingerprint image is compared with a stored fingerprint template to determine that the finger tapping the touchscreen is the index finger (1) of the right hand, and the fingertip is the fingertip ○. Then, the area tapped on the touchscreen is determined to be the right side area. Figure 2 From the symbol mapping relationship in the diagram, we can see that the corresponding output symbol is W.

[0123] Furthermore, different types of symbolic inputs can have their own independent symbolic mapping relationships. For example... Figure 2 The symbol mapping relationship between letters and numbers is shown. When the current symbol input interface is a letter symbol input interface, the required letter symbol needs to be matched according to the symbol mapping relationship. For example, the fingerprint image is compared with the stored fingerprint template, and the finger information is identified as follows: the finger that the current user taps the touch screen is the middle finger of the right hand (2), and the finger part that taps the touch screen is the fingertip ⌒. The tapping position information is obtained as: the right area. The current input mode is letter input, so it can be obtained through... Figure 2 The symbol mapping relationship shown indicates that the corresponding symbol is the letter L.

[0124] Alternatively, when the current symbol input interface is a numeric input interface, the fingerprint image is compared with the stored fingerprint template to identify the finger information as follows: the finger that the user is tapping the touchscreen is the middle finger of the right hand (2), and the finger part tapping the touchscreen is the fingertip ⌒. The tapping position information is obtained as: the left area, currently in numeric input mode, then it can be obtained through Figure 2 The symbol mapping relationship of the shown numbers determines that the corresponding symbol is the number 7.

[0125] In one possible implementation, after identifying the finger information and the tapping location information, the method further includes:

[0126] The tapping area is determined based on the finger information and the tapping location information;

[0127] The striking area is matched with a predefined symbol mapping table to obtain the symbol corresponding to the striking area;

[0128] The symbol corresponding to the tapping area is output to the display device.

[0129] In this embodiment, a corresponding symbol mapping table is generated based on the tapping area and the symbol. In practical applications, the tapping area is determined based on the finger information and tapping position information identified from the fingerprint image. The tapping area is then matched with the symbol mapping table to obtain the corresponding symbol, which is then displayed on the display device.

[0130] In addition, to achieve input error correction and further improve the accuracy of symbol input, statistical models of natural language can be used to correct the combination of input symbols.

[0131] Step S104: The feedback module generates visual feedback based on the symbol and displays the symbol on the display screen.

[0132] In this embodiment, the finger positioning module determines the type of one or more tapping fingers, the finger part of the tapping finger, and the sensor area of ​​the tapping finger based on the image currently collected by the sensor, maps the symbol, and displays the symbol on the display screen through the feedback module, thereby realizing visual feedback.

[0133] In one possible implementation, the method further includes:

[0134] Through the feedback module, sound feedback is generated based on the symbols, and / or,

[0135] Vibration feedback is generated based on the symbol through the feedback module.

[0136] In this embodiment, the feedback module can provide feedback on symbol input through various means such as vision, sound, and touch. Specifically, visual feedback is generated by displaying the mapped symbol on the screen; sound feedback is generated by synchronously playing the pronunciation of the symbol; and vibration feedback is generated by synchronously vibrating each time a symbol is successfully output. Thus, by using one or more of these feedback methods, the efficiency of symbol input is further improved, and the user's reliance on visual input while typing is reduced.

[0137] In one possible implementation, the symbol input system further includes: a fingerprint registration module; the method further includes:

[0138] For first-time registered users, the fingerprint registration module activates the fingerprint sensor to acquire registration fingerprint images of different fingers of the registered user; the registration fingerprint image is a complete rolling fingerprint image for a single finger, and / or a planar fingerprint image for each part of the finger for a single finger;

[0139] Fingerprint features are extracted from the registered fingerprint image to generate fingerprint templates for each finger of the registered user;

[0140] The fingerprint templates of all the registered user's fingers are stored in the fingerprint database.

[0141] In this embodiment, before a user inputs a symbol for the first time, they need to complete registration to store their fingerprint data. Specifically, refer to... Figure 3 , Figure 3 A flowchart illustrating a fingerprint registration process is shown, such as... Figure 3 As shown, after the fingerprint registration module receives a registration request, it acquires fingerprint images for different fingers of the registered user to obtain registration fingerprint images. During this process, different symbol mapping relationships impose different requirements on the acquisition of registration fingerprint images. Specifically, if the pre-set symbol mapping relationship is only for the finger type level and does not involve the mapping relationship for the finger part that taps the touchscreen, then during image acquisition, it is only necessary to acquire the complete scrolling fingerprint image for each finger of the user. If the symbol mapping relationship also includes mapping relationships between different finger parts and symbols, then during image acquisition, it is necessary to acquire planar fingerprint images of each finger part, for example, the fingerprint image of the fingertip of the right index finger and the fingerprint image of the fingertip of the right index finger.

[0142] like Figure 3 As shown, after image acquisition is completed, the acquired registered fingerprint image needs to be preprocessed through operations such as enhancement, cropping, and correction.

[0143] In one possible implementation, after acquiring the registered fingerprint image, the method further includes:

[0144] The fingerprint quality of the registered fingerprint image is assessed to determine its quality.

[0145] If the fingerprint quality of the registered fingerprint image is lower than a preset quality threshold, a new registered fingerprint image of that finger is acquired. Specifically, during fingerprint registration, a fingerprint quality judgment function can be added to automatically identify poor-quality fingerprint images (registered fingerprint images with fingerprint quality lower than a preset quality threshold) and remind the user to re-register that finger, thereby improving the quality of the registered fingerprint template.

[0146] In this embodiment, feature extraction is performed on the obtained registration fingerprint image to obtain fingerprint features. Specifically, feature extraction algorithms can be used, such as one or more of minutiae feature extraction algorithms, ridge and valley feature extraction algorithms, and local feature extraction algorithms, to extract fingerprint features (orientation field, period, endpoint, bifurcation point, etc., or other machine learning depth features) from the registration fingerprint image, generate the fingerprint template of the registered user, and store it in the fingerprint database.

[0147] Through the above registration steps, the fingerprint database stores the fingerprint templates of all authorized users. After the current fingerprint image is acquired, the fingerprint image is matched one by one with the fingerprint templates in the fingerprint database. If no matching fingerprint template is found, it means that the user is not an authorized user. This achieves user authentication during the symbol input process and further improves system security.

[0148] Reference Figure 4 , Figure 4 A schematic diagram of a symbol input process based on fingerprint recognition is shown, such as... Figure 4 As shown, when a user uses the symbol input system for the first time, they first register their fingerprint using the fingerprint registration module. The fingerprint registration module controls the fingerprint sensor to collect the user's registered fingerprint image, extracting the fingerprint features of each typing finger to generate a fingerprint template. After registration, when the user needs to input symbols, the fingerprint sensor collects a fingerprint image. The finger positioning module performs image preprocessing and feature extraction to obtain the fingerprint features of one or more striking fingerprints. These fingerprint features are matched with fingerprint templates in the fingerprint database to determine the finger type in the fingerprint image and the finger position on the touchscreen, as well as the area of ​​the touchscreen being struck (striking location information). Then, based on this information, the symbol mapping module performs symbol mapping according to a pre-defined symbol mapping relationship to obtain the symbol. Finally, the feedback module generates visual, auditory, and vibration feedback based on the symbol. Therefore, this embodiment of the application, by recognizing the striking fingers and based on a predefined symbol mapping relationship, achieves a technical solution for finger symbol input (i.e., blind typing) and command control without needing to look at the keyboard. Compared with the prior art, the symbol input method proposed in this application is more natural and intuitive to operate, and can customize symbol mapping to achieve personalized input, solving problems such as visual dependence, low efficiency and lack of personalized input. At the same time, it can also assist in identity recognition and improve the security of use.

[0149] The second aspect of this application also proposes a symbol input system based on fingerprint recognition, referring to... Figure 5 , Figure 5 A schematic diagram of a symbol input system is shown, such as... Figure 5 As shown, the system includes:

[0150] The fingerprint image acquisition module is used to acquire fingerprint images of the user's finger tapping on the touch screen;

[0151] The finger positioning module is used to compare the acquired fingerprint image with the stored fingerprint template to identify the finger information and tapping position information of the fingerprint image;

[0152] The symbol mapping module is used to map the finger information and the tapping position information into symbols according to a predefined symbol mapping relationship;

[0153] A feedback module is used to generate visual feedback based on the symbol and display the symbol on a display screen.

[0154] In one possible implementation, the finger positioning module includes:

[0155] The finger type determination submodule is used to match the fingerprint image with the stored fingerprint templates one by one to determine the finger type of the user's finger in the fingerprint image; the finger type includes at least: left hand, right hand, index finger, ring finger, little finger, thumb, and middle finger;

[0156] The finger location determination submodule is used to determine the finger location to which the fingerprint image belongs. The finger location includes at least: fingertip, finger pad, left finger side, and right finger side.

[0157] In one possible implementation, the system further includes: a fingerprint registration module, used for:

[0158] For first-time registered users, the fingerprint sensor is activated to acquire registered fingerprint images of different fingers of the registered user; the registered fingerprint image is a complete rolling fingerprint image for a single finger, and / or a planar fingerprint image for each part of the finger for a single finger;

[0159] Fingerprint features are extracted from the registered fingerprint image to generate fingerprint templates for each finger of the registered user;

[0160] The fingerprint templates of all the registered user's fingers are stored in the fingerprint database.

[0161] In one possible implementation, the system further includes:

[0162] The quality assessment module is used to assess the fingerprint quality of the registered fingerprint image after it has been acquired, and to determine the fingerprint quality of the registered fingerprint image.

[0163] The reminder module is used to re-acquire the registered fingerprint image of the finger if the fingerprint quality of the registered fingerprint image is lower than a preset quality threshold.

[0164] In one possible implementation, the finger positioning module includes:

[0165] The first tapping location information acquisition submodule is used to acquire the area information corresponding to the touch screen tapped by the user's finger in the fingerprint image when the fingerprint sensor is a large-area sensor that can accommodate multiple fingers, and the touch screen of the fingerprint sensor is divided into multiple areas, and to use the area information as the tapping location information.

[0166] The second tapping location information acquisition submodule is used to acquire the identity information of the small-area sensor that collects the fingerprint image when the fingerprint sensor is a combination of multiple small-area sensors that can only accommodate a single finger, and use the identity information of the small-area sensor as the tapping location information.

[0167] In one possible implementation, the system further includes:

[0168] The first symbol mapping relationship generation module is used to generate the symbol mapping relationship by performing symbol mapping according to the finger type and finger location when the fingerprint sensor is a small-area sensor that only supports output from a single finger.

[0169] The second symbol mapping relationship generation module is used to divide the touch screen of the fingerprint sensor into multiple areas when the fingerprint sensor is a large-area sensor that can accommodate multiple fingers, and perform symbol mapping based on finger type, finger position and different areas of the touch screen to generate the symbol mapping relationship.

[0170] The third symbol mapping relationship generation module is used to generate the symbol mapping relationship by combining different combinations of finger type, finger location and the small area sensors when the fingerprint sensor is a combination of multiple small area sensors that can only accommodate a single finger.

[0171] In one possible implementation, the finger positioning module is further configured to extract features from the fingerprint image using a feature extraction algorithm to obtain fingerprint features; the feature extraction algorithm is at least one of minutiae feature extraction algorithm, local feature extraction algorithm, ridge feature extraction algorithm, and deep learning descriptor extraction algorithm.

[0172] The finger positioning module is further configured to use a matching algorithm to match the fingerprint features with fingerprint templates stored in the fingerprint database. The matching algorithm is at least one of sliding window search, correlation matching, feature point matching, and similarity distance judgment.

[0173] In one possible implementation, the feedback module is further configured to generate sound feedback based on the symbol, and / or generate vibration feedback based on the symbol.

[0174] In one possible implementation, the finger positioning module is further configured to, when the fingerprint image includes multiple fingerprints, split the fingerprint image into multiple independent fingerprint images, match each independent fingerprint image with a fingerprint template stored in a fingerprint database, and obtain finger information and tapping position information in each independent fingerprint image.

[0175] In one possible implementation, the device further includes:

[0176] The tapping area determination module is used to determine the tapping area based on the finger information and the tapping position information after the finger information and tapping position information are identified.

[0177] The matching module is used to match the tapping area with a predefined symbol mapping table to obtain the symbol corresponding to the tapping area;

[0178] The display module is used to output the symbol corresponding to the tapping area to the display device.

[0179] This application also provides an electronic device, see embodiments thereof. Figure 6 , Figure 6 This is a schematic diagram of the electronic device proposed in an embodiment of this application. Figure 6 As shown, the electronic device 100 includes a memory 110 and a processor 120. The memory 110 and the processor 120 are connected via a bus. The memory 110 stores a computer program that can run on the processor 120 to implement the steps in the symbol input method based on fingerprint recognition disclosed in the embodiments of this application.

[0180] This application also provides a computer-readable storage medium storing a computer program / instructions thereon, which, when executed by a processor, implements the steps in the symbol input method based on fingerprint recognition disclosed in this application.

[0181] This application also provides a computer program product that, when run on an electronic device, causes a processor to execute the steps of the symbol input method based on fingerprint recognition as disclosed in this application.

[0182] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0183] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, electronic devices, and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0184] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0185] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0186] Although preferred embodiments of the present application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present application.

[0187] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0188] The above provides a detailed description of a symbol input method based on fingerprint recognition provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A symbol input method based on fingerprint recognition, characterized in that, Applied to a symbol input system, the symbol input system includes: a fingerprint image acquisition module, a finger positioning module, a symbol mapping module, and a feedback module; the method includes: The fingerprint image acquisition module acquires fingerprint images of the user's finger tapping on the touchscreen. Using the finger positioning module, the acquired fingerprint image is compared with the stored fingerprint template to identify the finger information and tapping position information of the fingerprint image; The symbol mapping module maps the finger information and the tapping position information into symbols according to a predefined symbol mapping relationship. The feedback module generates visual feedback based on the symbol and displays the symbol on the screen. The step of using the finger positioning module to compare the acquired fingerprint image with a stored fingerprint template to identify the finger information of the fingerprint image includes: The fingerprint image is matched one by one with the stored fingerprint templates using the finger positioning module to determine the finger type of the user's finger in the fingerprint image; the finger type includes at least: left hand, right hand, index finger, ring finger, little finger, thumb, and middle finger; The finger part to which the fingerprint image belongs is determined, and the finger part includes at least: fingertip, finger pad, left finger side, and right finger side; The tapping location information of the fingerprint image is obtained through the following steps: When the fingerprint image acquisition module is a large-area sensor that can accommodate multiple fingers, the touch screen of the fingerprint image acquisition module is divided into multiple areas, and the area information corresponding to the touch screen tapped by the user's finger in the fingerprint image is obtained, and the area information is used as the tapping position information; When the fingerprint image acquisition module is a combination of multiple small-area sensors that can only accommodate a single finger, the identity information of the small-area sensor that acquires the fingerprint image is obtained, and the identity information of the small-area sensor is used as the tapping position information.

2. The symbol input method based on fingerprint recognition according to claim 1, characterized in that, The symbol input system further includes: a fingerprint registration module; the method further includes: For first-time registered users, the fingerprint registration module activates the fingerprint image acquisition module to acquire registration fingerprint images of different fingers of the registered user; the registration fingerprint image is a complete rolling fingerprint image of a single finger, and / or a planar fingerprint image of each finger portion of a single finger; Fingerprint features are extracted from the registered fingerprint image to generate fingerprint templates for each finger of the registered user; The fingerprint templates of all the registered user's fingers are stored in the fingerprint database.

3. The symbol input method based on fingerprint recognition according to claim 2, characterized in that, After collecting registration fingerprint images of different fingers of the registered user, the method further includes: The fingerprint quality of the registered fingerprint image is assessed to determine its quality. If the fingerprint quality of the registered fingerprint image is lower than a preset quality threshold, the registered fingerprint image of that finger is re-acquired.

4. The symbol input method based on fingerprint recognition according to claim 1, characterized in that, The method further includes: When the fingerprint image acquisition module is a small-area sensor that only supports output from a single finger, symbol mapping is performed according to the finger type and finger location to generate the symbol mapping relationship; When the fingerprint image acquisition module is a large-area sensor that can accommodate multiple fingers, the touch screen of the fingerprint image acquisition module is divided into multiple areas, and symbol mapping is performed by combining the finger type, finger position and different areas of the touch screen to generate the symbol mapping relationship; When the fingerprint image acquisition module is a combination of multiple small-area sensors that can only accommodate a single finger, symbol mapping is performed by combining the finger type, finger location, and different combinations of the small-area sensors to generate the symbol mapping relationship.

5. The symbol input method based on fingerprint recognition according to claim 2, characterized in that, The step of matching the fingerprint image with the stored fingerprint templates one by one includes: The finger positioning module uses a feature extraction algorithm to extract features from the fingerprint image to obtain fingerprint features; the feature extraction algorithm is at least one of the following: minutiae feature extraction algorithm, local feature extraction algorithm, ridge feature extraction algorithm, and deep learning descriptor extraction algorithm. The finger positioning module uses a matching algorithm to match the fingerprint features with fingerprint templates stored in the fingerprint database. The matching algorithm is at least one of the following: sliding window search method, correlation matching method, feature point matching method, and similarity distance judgment method.

6. The symbol input method based on fingerprint recognition according to claim 1, characterized in that, The method further includes: Through the feedback module, sound feedback is generated based on the symbols, and / or, Vibration feedback is generated based on the symbol through the feedback module.

7. The symbol input method based on fingerprint recognition according to claim 2, characterized in that, The method further includes: When the fingerprint image includes multiple fingerprints, the fingerprint image is split into multiple independent fingerprint images. Using the finger positioning module, each independent fingerprint image is matched with a fingerprint template stored in the fingerprint database to obtain the finger information and tapping position information in each independent fingerprint image.

8. The symbol input method based on fingerprint recognition according to claim 1, characterized in that, After identifying the finger information and the tapping location information, the method further includes: The tapping area is determined based on the finger information and the tapping location information; The striking area is matched with a predefined symbol mapping table to obtain the symbol corresponding to the striking area; The symbol corresponding to the tapping area is output to the display device.

Images