Fingerprint-based input interaction method and device

Abstract

The invention provides an input interaction method and device based on fingerprints, and relates to the technical field of man-machine interaction. The method comprises the following steps: acquiring continuous fingerprint frame images when a user uses a fingerprint sensor to perform input interaction with intelligent equipment; performing identity authentication on the user based on the continuous fingerprint frame image and pre-registered user fingerprint information; when the identity authentication of the user is passed, recognizing the action of the touch operation corresponding to each fingerprint sensor according to the change information of the plurality of continuous fingerprint frame images corresponding to each fingerprint sensor in the time sequence; and determining an input interaction instruction matched with the input interaction according to the application scene and the action information of the intelligent equipment. According to the invention, identity authentication and action recognition are carried out on the user, so that the security of the input interaction process of the user is improved. Multiple actions can be recognized based on the change information of the continuous fingerprint frame images in the time sequence, input interaction modes are enriched, and richer, more convenient and more personalized interaction experience is provided for users.

Description

Technical Field

[0001] The present disclosure relates to the field of human-computer interaction technology, and in particular to a fingerprint-based input interaction method and device. Background Art

[0002] With the rise of AR / VR technology and the popularity of wearable electronic products, people are increasingly demanding more convenient and efficient ways of human-computer interaction. Fingerprint sensors, as a classic biometric sensor, are usually used for identity authentication. In related technologies, there are solutions that use fingerprint sensors for motion recognition and then human-computer interaction. At present, smart devices such as smart glasses and in-ear headphones mostly use narrow and long touch input strips, which can realize simple interaction technologies such as pressing. The interaction method is single and cannot support richer input interaction functions. Summary of the invention

[0003] The present disclosure aims to solve one of the technical problems in the related art at least to some extent.

[0004] To this end, the first embodiment of the present disclosure proposes a fingerprint-based input interaction method, which is applied to a smart device and includes the following steps:

[0005] In response to a user using at least one fingerprint sensor to perform input interaction with the smart device, acquiring a plurality of continuous fingerprint frame images corresponding to each fingerprint sensor when the user performs a touch operation on the at least one fingerprint sensor;

[0006] authenticating the user based on a plurality of continuous fingerprint frame images corresponding to one or more fingerprint sensors of the at least one fingerprint sensor and pre-registered user fingerprint information;

[0007] In response to the user's identity authentication being passed, determining whether the touch operation corresponding to each fingerprint sensor is a click action type according to change information of a plurality of continuous fingerprint frame images corresponding to each fingerprint sensor in a time series;

[0008] For a first fingerprint sensor of the at least one fingerprint sensor whose touch operation is a click action type, extracting contact time variation information from a plurality of continuous fingerprint frame images corresponding to the first fingerprint sensor, and determining a first action corresponding to the first fingerprint sensor according to the contact time variation information;

[0009] For a second fingerprint sensor whose touch operation is a non-click action type among the at least one fingerprint sensor, extract total displacement information, center of gravity displacement information, and angle change information of fingerprint features in a plurality of continuous fingerprint frame images corresponding to the second fingerprint sensor, and determine a second action corresponding to the second fingerprint sensor according to the total displacement information, the center of gravity displacement information, and the angle change information;

[0010] An input interaction instruction matching the input interaction is determined according to the application scenario and action information of the smart device, where the action information includes the first action and / or the second action.

[0011] In some embodiments of the present disclosure, determining the second action corresponding to the second fingerprint sensor based on the total displacement information, the center of gravity displacement information and the angle change information includes: in response to the total displacement information being greater than a first threshold, the center of gravity displacement information being less than or equal to a second threshold, and the angle change information being less than or equal to a third threshold, determining that the second action is a sliding action; or, in response to the total displacement information being less than or equal to the first threshold, the center of gravity displacement information being greater than the second threshold, and the angle change information being less than or equal to the third threshold, determining that the second action is a rolling action; or, in response to the total displacement information being less than or equal to the first threshold, the center of gravity displacement information being less than or equal to the second threshold, and the angle change information being greater than the third threshold, determining that the second action is a rotation action.

[0012] In some embodiments of the present disclosure, determining the second action corresponding to the second fingerprint sensor based on the total displacement information, the center of gravity displacement information and the angle change information includes: determining the second action corresponding to the second fingerprint sensor through a pre-trained classifier based on the total displacement information, the center of gravity displacement information and the angle change information; wherein the classifier learns through a machine learning method to obtain the mapping relationship between the total displacement information, the center of gravity displacement information and the angle change information and the action.

[0013] In some embodiments of the present disclosure, determining the input interaction instruction that matches the input interaction based on the application scenario and action information of the smart device includes: identifying the finger information of the touch operation on each fingerprint sensor based on multiple continuous fingerprint frame images corresponding to each fingerprint sensor and pre-registered user fingerprint information; determining the mapping relationship between the action, finger and input interaction instruction based on the application scenario; and determining the input interaction instruction that matches the input interaction based on the action information, the finger information and the mapping relationship.

[0014] In some embodiments of the present disclosure, the input interaction instructions include action execution instructions and / or symbol input instructions of the smart device.

[0015] In some embodiments of the present disclosure, the method further includes: generating interactive feedback information based on the touch operation, the interactive feedback information is used to provide touch feedback to the user, and the interactive feedback information includes at least one of the following: image feedback information, sound feedback information and vibration feedback information.

[0016] In some embodiments of the present disclosure, the at least one fingerprint sensor is located in a side area of ​​the smart device.

[0017] The second aspect of the present disclosure provides a fingerprint-based input interaction device, which is provided in a smart device and is characterized by comprising:

[0018] an acquisition module, configured to acquire, in response to a user using at least one fingerprint sensor to perform input interaction with the smart device, a plurality of continuous fingerprint frame images corresponding to each fingerprint sensor when the user performs a touch operation on the at least one fingerprint sensor;

[0019] an identity authentication module, configured to authenticate the user based on a plurality of continuous fingerprint frame images corresponding to one or more fingerprint sensors of the at least one fingerprint sensor and pre-registered user fingerprint information;

[0020] A first determination module is configured to determine, in response to the user's identity authentication being passed, whether the touch operation corresponding to each fingerprint sensor is a click action type according to change information of a plurality of continuous fingerprint frame images corresponding to each fingerprint sensor in a time series;

[0021] a second determining module, configured to extract, for a first fingerprint sensor whose touch operation is a click action type among the at least one fingerprint sensor, contact time variation information from a plurality of continuous fingerprint frame images corresponding to the first fingerprint sensor, and determine a first action corresponding to the first fingerprint sensor according to the contact time variation information;

[0022] a third determination module, configured to extract, for a second fingerprint sensor whose touch operation is a non-click action type among the at least one fingerprint sensor, total displacement information, center of gravity displacement information, and angle change information of fingerprint features in a plurality of continuous fingerprint frame images corresponding to the second fingerprint sensor, and determine a second action corresponding to the second fingerprint sensor according to the total displacement information, the center of gravity displacement information, and the angle change information;

[0023] The fourth determination module is used to determine an input interaction instruction matching the input interaction according to the application scenario and action information of the smart device, where the action information includes the first action and / or the second action.

[0024] A third aspect of the present disclosure provides an intelligent device, comprising: a processor, and a memory communicatively connected to the processor;

[0025] The memory stores computer-executable instructions;

[0026] The processor executes the computer-executable instructions stored in the memory to implement the method described in the first aspect.

[0027] The fourth aspect of the present disclosure provides a computer-readable storage medium, characterized in that the computer-readable storage medium stores computer execution instructions, and when the computer execution instructions are executed by a processor, they are used to implement the method described in the first aspect.

[0028] Through the fingerprint-based input interaction method provided by the embodiment of the present disclosure, by analyzing the continuous fingerprint frame images corresponding to at least one fingerprint sensor, the user's identity authentication and action recognition are performed, thereby improving the security of the user's input interaction process. Based on the change information of the continuous fingerprint frame images in the time series, a variety of actions can be identified, and action recognition based on a small-area fingerprint sensor can be realized, and the corresponding input interaction instructions can be further determined, enriching the input interaction mode, and providing users with a richer, more convenient, and personalized interaction experience.

[0029] Additional aspects and advantages of the present disclosure will be given in part in the following description and in part will be obvious from the following description or learned through practice of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The above and / or additional aspects and advantages of the present disclosure will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0031] Figure 1 A schematic diagram of a flow chart of a fingerprint-based input interaction method provided in an embodiment of the present disclosure;

[0032] Figure 2 A schematic diagram of a sliding action, a rolling action and a rotating action provided in an embodiment of the present disclosure;

[0033] Figure 3 A schematic diagram of continuous fingerprint frame images corresponding to a sliding action, a rolling action and a rotating action provided in an embodiment of the present disclosure;

[0034] Figure 4A schematic diagram of a mapping relationship between an action, a finger and a symbol input instruction provided in an embodiment of the present disclosure;

[0035] Figure 5 A schematic diagram of a fingerprint-based input interaction device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0036] Embodiments of the present disclosure are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present disclosure, and should not be construed as limiting the present disclosure.

[0037] The present disclosure provides a fingerprint-based input interaction method and device. Specifically, the fingerprint-based input interaction method and device of the present disclosure are described below with reference to the accompanying drawings.

[0038] Figure 1 The present invention provides a schematic flow chart of a fingerprint-based input interaction method provided in an embodiment of the present invention, and the input interaction method is applied to a smart device. Figure 1 As shown, the fingerprint-based input interaction method may include but is not limited to the following steps:

[0039] Step 101, in response to a user using at least one fingerprint sensor to perform input interaction with a smart device, a plurality of continuous fingerprint frame images corresponding to each fingerprint sensor when the user performs a touch operation on the at least one fingerprint sensor are acquired.

[0040] In the embodiments of the present disclosure, a user may use one fingerprint sensor to perform input interaction with a smart device, or may use multiple fingerprint sensors to perform input interaction with a smart device.

[0041] The fingerprint sensor is one or more of the following: an optical sensor, an ultrasonic sensor, and a capacitive sensor. The fingerprint sensor may be a narrow strip fingerprint sensor, or a fingerprint sensor of other shapes, such as a circular or square fingerprint sensor with a larger area. The smart device may be a wearable smart device such as smart glasses, in-ear headphones, smart watches, or may also be a smart device such as a mobile phone or a tablet computer.

[0042] Optionally, at least one fingerprint sensor in the embodiment of the present disclosure may be located in the side area of ​​the smart device. For example, one or more fingerprint sensors may be set on the side of the frame or the side area of ​​the temple of the smart glasses.

[0043] In order to optimize the user experience, optionally, in some embodiments of the present disclosure, when the user performs a touch operation on the fingerprint sensor, the smart device may generate interactive feedback information based on the touch operation, and the interactive feedback information is used to provide touch feedback to the user, and the interactive feedback information includes at least one of the following: image feedback information, sound feedback information and vibration feedback information.

[0044] In some embodiments of the present disclosure, when each fingerprint sensor collects continuous fingerprint frame images of the user, the collected original images can be preprocessed in advance, such as conversion to grayscale, Gaussian blur processing, image enhancement, cropping, correction, etc.

[0045] Step 102: Authenticate the user based on a plurality of continuous fingerprint frame images corresponding to one or more fingerprint sensors of at least one fingerprint sensor and pre-registered user fingerprint information.

[0046] In one implementation, the user's fingerprint information can be pre-registered when the user uses the smart device for the first time or enables fingerprint permission for the first time. The user can slide one or more fingers horizontally across the fingerprint sensor to record the sequence frame image of each registered finger fingerprint, extract the complete fingerprint features (such as the details points and line direction in the fingerprint image) based on the sequence frame image, and generate and store the user's user fingerprint information. During registration, a fingerprint quality judgment function can also be added to automatically identify poor fingerprint images and remind the user to re-register the finger to improve the quality of the registered fingerprint template.

[0047] When a user interacts with a smart device through input, the user's identity can be authenticated based on the pre-registered user fingerprint information and the continuous fingerprint frame images collected during the current interaction process to ensure the security of human-computer interaction and the user's personal privacy.

[0048] Step 103, in response to the user's identity authentication being passed, determining whether the touch operation corresponding to each fingerprint sensor is a click action type according to the change information of a plurality of continuous fingerprint frame images corresponding to each fingerprint sensor in a time series.

[0049] As an example, in the current touch operation, the action where the contact time is less than the touch time threshold and / or the fingerprint features in multiple consecutive fingerprint frame images have no obvious displacement change, center of gravity change, or angle change can be determined as a click action type, and the action where the contact time is greater than or equal to the touch time threshold and the fingerprint features have obvious changes can be determined as a non-click action. Among them, whether there is an obvious change can be confirmed by comparing with a preset threshold.

[0050] Taking the single-click action in the click action type as an example, it can be confirmed based on the continuous fingerprint frame images that one touch occurred in the touch operation, and the time the user was in contact with the fingerprint sensor was less than the touch time threshold. Taking the double-click action as an example, it can be confirmed based on the continuous fingerprint frame images that two touches occurred in the touch operation, and the time the user was in contact with the fingerprint sensor twice was less than the touch time threshold. Taking the single long press action as an example, it can be confirmed based on the continuous fingerprint frame images that one touch occurred in the touch operation. Although the contact time is greater than or equal to the touch time threshold, by comparing the feature changes between the continuous fingerprint frame images, it is found that there is no obvious displacement change, center of gravity change, or angle change in this touch operation. Taking the combination action of single click and long press as an example, it can be confirmed based on the continuous fingerprint frame images that two touches occurred in the touch operation. If the contact time of the two touches is less than the touch time threshold, it is a single click. If the contact time is greater than or equal to the touch time threshold but there is no obvious change in the fingerprint feature, it is a long press. Since single click and long press both belong to the click action type, it can be determined that the combination action of single click and long press is a click action type.

[0051] Step 104: for a first fingerprint sensor whose touch operation is a click action type among at least one fingerprint sensor, extract contact time variation information from a plurality of continuous fingerprint frame images corresponding to the first fingerprint sensor, and determine a first action corresponding to the first fingerprint sensor according to the contact time variation information.

[0052] Based on the contact time change information in multiple consecutive fingerprint frame images, it can be confirmed how many clicks have occurred in the current touch operation, whether each click is a single click or a long press, thereby confirming the first action corresponding to the first fingerprint sensor (such as a single click, a long press, or a combination of a single click and a long press).

[0053] Step 105, for a second fingerprint sensor whose touch operation is a non-click action type in at least one fingerprint sensor, extract total displacement information, center of gravity displacement information, and angle change information of fingerprint features in multiple continuous fingerprint frame images corresponding to the second fingerprint sensor, and determine a second action corresponding to the second fingerprint sensor based on the total displacement information, center of gravity displacement information, and angle change information.

[0054] Optionally, in some embodiments of the present disclosure, three non-click actions are provided, namely, a sliding action, a rolling action and a rotating action. Figure 2A schematic diagram of a sliding action, a rolling action and a rotating action provided in an embodiment of the present disclosure. It should be noted that the total displacement information represents the total displacement of the finger relative to the smart device or the fingerprint sensor when the user performs a touch operation, and the total displacement information may include information such as the total displacement distance, the total displacement direction, and the total displacement speed. The center of gravity in the center of gravity displacement information represents the center of gravity of the fingerprint pattern or fingerprint feature in the fingerprint frame image, and the center of gravity displacement information may include information such as the center of gravity displacement distance, the center of gravity displacement direction, and the center of gravity displacement speed. The angle change information includes the angle change of the fingerprint feature in the fingerprint frame image, and the angle change information may include information such as the angle change amount, the angle change direction, and the angular velocity.

[0055] In one implementation, the second action may be determined based on the total displacement information, the center of gravity displacement information, the angle change information, and their corresponding thresholds. Figure 3 A schematic diagram of continuous fingerprint frame images corresponding to a sliding action, a rolling action, and a rotating action provided by an embodiment of the present disclosure. As an example, when the total displacement information is greater than the first threshold, the center of gravity displacement information is less than or equal to the second threshold, and the angle change information is less than or equal to the third threshold, the second action can be determined to be a sliding action. Figure 3 As shown, when the finger slides (corresponding to Figure 2 The second action is determined to be a rolling action. When the finger performs a rolling action, the fingerprint pattern in the continuous fingerprint frame images is obviously displaced in the fingerprint collection area, but the finger itself does not obviously move or rotate, and the same fingerprint features in multiple continuous fingerprint frame images do not have obvious displacement. When the total displacement information is less than or equal to the first threshold, the center of gravity displacement information is less than or equal to the second threshold, and the angle change information is greater than the third threshold, the second action is determined to be a rotational action. When the finger performs a rotational action, the angle of the fingerprint pattern in the continuous fingerprint frame images changes significantly, but the finger itself and the center of gravity of the pattern in the fingerprint collection area do not change significantly.

[0056] In another implementation, the second action can also be identified by a pre-trained classifier. As an example, the total displacement information, the center of gravity displacement information, and the angle change information can be input into a pre-trained classifier to determine the second action corresponding to the second fingerprint sensor. The classifier obtains the mapping relationship between the total displacement information, the center of gravity displacement information, and the angle change information and the action through machine learning methods (such as support vector machines, neural networks, etc.) or empirical algorithms to identify the user action.

[0057] Optionally, the fingerprint feature may include any one of a fingerprint detail point feature, a local feature, a ridge feature, and a deep learning descriptor.

[0058] It should be noted that since the fingerprint-based input interaction method proposed in the embodiment of the present disclosure recognizes different user actions based on the feature changes of continuous fingerprint frame images in a time series, it does not require a high area of ​​the fingerprint sensor. It is therefore suitable for action recognition of sensors with smaller areas, such as narrow strip-shaped fingerprint sensors. Sensors with larger areas can also use the method proposed in this embodiment.

[0059] Step 106: Determine an input interaction instruction that matches the input interaction according to the application scenario and action information of the smart device, where the action information includes the first action and / or the second action.

[0060] The input interaction instructions include action execution instructions and / or symbol input instructions of the smart device.

[0061] It should be noted that in different application scenarios, the same touch operation on the same smart device triggers different interactive instructions. Taking smart glasses as an example, in the music playback scenario, the following action execution instructions can be confirmed by touching the fingerprint sensor:

[0062] Scroll left and right: fast forward / rewind;

[0063] Pan up and down: increase volume / decrease volume;

[0064] Swipe left or right: switch to the previous song / switch to the next song.

[0065] In the map navigation scenario, the same touch operation can correspond to the following action execution instructions:

[0066] Scroll left and right: zoom in / out the map;

[0067] Up and down translation: the map moves up and down;

[0068] Move left and right: the map moves left and right;

[0069] Rotate left and right: The map rotates.

[0070] It should be noted that the symbol input instruction refers to the symbol input interaction achieved through the user's touch operation on the fingerprint sensor. Through the mapping relationship between symbols and actions, the user can input symbols without relying on vision, only through the interaction between the finger and the fingerprint sensor.

[0071] It should also be noted that when the number of fingerprint sensors is one, the first action or the second action corresponding to the fingerprint sensor can be determined according to the user's touch operation on a single fingerprint sensor, and then the input interaction instruction can be determined. When the number of fingerprint sensors is two, the action corresponding to each fingerprint sensor is determined according to the user's touch operation on the two fingerprint sensors. Both fingerprint sensors can be the first action (such as the left hand finger clicks the left fingerprint sensor in the smart glasses, and the right hand finger clicks the right fingerprint sensor), or both can be the second action (for example, both fingerprint sensors correspond to the forward sliding action), or can also be a combination of the first action and the second action (for example, one fingerprint sensor is a single click action, and the other fingerprint sensor is a rotation action). When the number of fingerprint sensors is more than two, similarly, multiple fingerprint sensors can have multiple action combinations. By setting the mapping relationship between the action and the input interaction instruction, corresponding instructions can be set for different actions (or action combinations). Users can customize the instructions corresponding to different actions to meet the user's personal habits, making the interaction process more personalized.

[0072] In order to further enrich the interaction mode, optionally, in some embodiments of the present disclosure, the finger information of the touch operation on each fingerprint sensor can be identified based on multiple continuous fingerprint frame images corresponding to each fingerprint sensor and the pre-registered user fingerprint information. Determine the mapping relationship between actions, fingers and input interaction instructions based on the application scenario. Determine the input interaction instruction that matches the input interaction based on the action information, finger information and the mapping relationship. In this way, it is determined which finger of the user has made what action and which input interaction instruction corresponds to it. Among them, the mapping relationship between actions, fingers and input interaction instructions can be customized by the user to meet personalized needs. Optionally, in the embodiments of the present disclosure, when identifying the finger information and the area where the fingerprint features in the image are located in the complete fingerprint based on the continuous fingerprint frame image, but not limited to the following matching algorithms can be used: sliding window search method, correlation matching method, feature point matching method, similarity distance judgment.

[0073] Figure 4 Schematic diagram of a mapping relationship between an action, a finger, and a symbol input instruction provided by an embodiment of the present disclosure. Figure 4 As shown, when different fingers perform touch operations for different actions, corresponding symbols can be input, thereby replacing traditional button input, keyboard input or language input. The false touch rate is low, and there are no problems such as environmental noise and privacy protection in voice input interaction. It does not occupy smart device space or screen space, enriching the functions of fingerprint sensors and user interaction experience.

[0074] Optionally, in some embodiments of the present disclosure, after the smart device performs the corresponding operation according to the input interaction instruction, the interaction completion feedback information may be generated to feedback that the smart device has executed the input interaction instruction input by the user. When the input interaction instruction is an action execution instruction, the feedback information may be completed through interaction in the form of image, sound, vibration, etc. When the input interaction instruction is a symbol input instruction, the feedback information may be completed through interaction in the form of image (displaying symbols), sound (reading symbols), vibration, etc.

[0075] By implementing the disclosed embodiments, the user's identity authentication and action recognition are performed by analyzing the continuous fingerprint frame images corresponding to at least one fingerprint sensor, thereby improving the security of the user's input interaction process. Based on the change information of the continuous fingerprint frame images in the time series, a variety of actions can be identified, and action recognition based on a small-area fingerprint sensor can be realized, and the corresponding input interaction instructions can be further determined, enriching the human-computer interaction mode and providing users with a richer, more convenient and personalized interaction experience.

[0076] Figure 5 This is a schematic diagram of a fingerprint-based input interaction device provided by an embodiment of the present disclosure, and the device is arranged in a smart device. Figure 5 As shown, the fingerprint-based input interaction device includes: an acquisition module 501 , an identity authentication module 502 , a first determination module 503 , a second determination module 504 , a third determination module 505 and a fourth determination module 506 .

[0077] The acquisition module 501 is used to acquire a plurality of continuous fingerprint frame images corresponding to each fingerprint sensor when the user performs a touch operation on the at least one fingerprint sensor in response to the user using the at least one fingerprint sensor to perform input interaction with the smart device.

[0078] The identity authentication module 502 is used to authenticate the user based on a plurality of continuous fingerprint frame images corresponding to one or more fingerprint sensors of the at least one fingerprint sensor and pre-registered user fingerprint information.

[0079] The first determination module 503 is used to determine whether the touch operation corresponding to each fingerprint sensor is a click action type according to the change information of multiple continuous fingerprint frame images corresponding to each fingerprint sensor in time series in response to the user's identity authentication being passed.

[0080] The second determination module 504 is used to extract contact time change information from multiple continuous fingerprint frame images corresponding to the first fingerprint sensor of at least one fingerprint sensor whose touch operation is a click action type, and determine the first action corresponding to the first fingerprint sensor according to the contact time change information.

[0081] The third determination module 505 is used to extract the total displacement information, center of gravity displacement information and angle change information of the fingerprint features in multiple continuous fingerprint frame images corresponding to the second fingerprint sensor of at least one fingerprint sensor whose touch operation is a non-click action type, and determine the second action corresponding to the second fingerprint sensor based on the total displacement information, center of gravity displacement information and angle change information.

[0082] The fourth determination module 506 is used to determine the input interaction instruction matching the input interaction according to the application scenario and action information of the smart device, where the action information includes the first action and / or the second action.

[0083] In some embodiments of the present disclosure, the third determination module 505 is specifically used to: in response to the total displacement information being greater than the first threshold, the center of gravity displacement information being less than or equal to the second threshold, and the angle change information being less than or equal to the third threshold, determine that the second action is a sliding action; or, in response to the total displacement information being less than or equal to the first threshold, the center of gravity displacement information being greater than the second threshold, and the angle change information being less than or equal to the third threshold, determine that the second action is a scrolling action; or, in response to the total displacement information being less than or equal to the first threshold, the center of gravity displacement information being less than or equal to the second threshold, and the angle change information being greater than the third threshold, determine that the second action is a rotation action.

[0084] In some embodiments of the present disclosure, the third determination module 505 is specifically used to: determine the second action corresponding to the second fingerprint sensor through a pre-trained classifier based on the total displacement information, the center of gravity displacement information and the angle change information; wherein the classifier learns through a machine learning method to obtain the mapping relationship between the total displacement information, the center of gravity displacement information and the angle change information and the action.

[0085] In some embodiments of the present disclosure, the fourth determination module 506 is specifically used to: identify the finger information of the touch operation on each fingerprint sensor based on multiple continuous fingerprint frame images corresponding to each fingerprint sensor and pre-registered user fingerprint information; determine the mapping relationship between actions, fingers and input interaction instructions based on the application scenario; determine the input interaction instructions that match the input interaction based on the action information, finger information and the mapping relationship.

[0086] In some embodiments of the present disclosure, the input interaction instructions include action execution instructions and / or symbol input instructions of the smart device.

[0087] In some embodiments of the present disclosure, Figure 5 Based on the embodiment shown, the fingerprint-based input interaction device may further include a feedback module, wherein the feedback module is used to generate interactive feedback information based on the touch operation, the interactive feedback information is used to provide touch feedback to the user, and the interactive feedback information includes at least one of the following: image feedback information, sound feedback information, and vibration feedback information.

[0088] In some embodiments of the present disclosure, at least one fingerprint sensor is located in a side area of ​​the smart device.

[0089] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0090] In order to implement the above embodiments, the present disclosure also proposes a smart device, including: a processor, and a memory connected to the processor; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory to implement the method provided by the above embodiments. As an example, the smart device can be a wearable smart device such as smart glasses, in-ear headphones, smart watches, or can also be a smart device such as a mobile phone, a tablet computer, etc.

[0091] In order to implement the above embodiments, the present disclosure further proposes a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the methods provided by the above embodiments.

[0092] In order to implement the above embodiments, the present disclosure also proposes a computer program product, including a computer program, which implements the methods provided by the above embodiments when executed by a processor.

[0093] The collection, storage, use, processing, transmission, provision and disclosure of user personal information involved in this disclosure shall comply with the relevant laws and regulations and shall not violate public order and good morals.

[0094] In the description of the aforementioned embodiments, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present disclosure. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, unless they are contradictory.

[0095] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of the present disclosure, "plurality" means at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0096] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present disclosure includes additional implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present disclosure belong.

[0097] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute the instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.

[0098] It should be understood that the various parts of the present disclosure can be implemented in hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented in software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used to implement: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0099] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0100] In addition, each functional unit in each embodiment of the present disclosure may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0101] The storage medium mentioned above may be a read-only memory, a disk or an optical disk, etc. Although the embodiments of the present disclosure have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations of the present disclosure. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present disclosure.

Claims

1. A fingerprint-based input interaction method, the method is applied to a smart device, characterized in that: The following steps are involved: In response to a user using at least one fingerprint sensor to perform input interaction with the smart device, acquiring a plurality of continuous fingerprint frame images corresponding to each fingerprint sensor when the user performs a touch operation on the at least one fingerprint sensor; authenticating the user based on a plurality of continuous fingerprint frame images corresponding to one or more fingerprint sensors of the at least one fingerprint sensor and pre-registered user fingerprint information; In response to the user's identity authentication being passed, determining whether the touch operation corresponding to each fingerprint sensor is a click action type according to change information of a plurality of continuous fingerprint frame images corresponding to each fingerprint sensor in a time series; For a first fingerprint sensor of the at least one fingerprint sensor whose touch operation is a click action type, extracting contact time variation information from a plurality of continuous fingerprint frame images corresponding to the first fingerprint sensor, and determining a first action corresponding to the first fingerprint sensor according to the contact time variation information; For a second fingerprint sensor whose touch operation is a non-click action type among the at least one fingerprint sensor, extract total displacement information, center of gravity displacement information, and angle change information of fingerprint features in a plurality of continuous fingerprint frame images corresponding to the second fingerprint sensor, and determine a second action corresponding to the second fingerprint sensor according to the total displacement information, the center of gravity displacement information, and the angle change information; An input interaction instruction matching the input interaction is determined according to the application scenario and action information of the smart device, where the action information includes the first action and / or the second action.

2. The method according to claim 1, characterized in that The determining, according to the total displacement information, the center of gravity displacement information, and the angle change information, a second action corresponding to the second fingerprint sensor includes: In response to the total displacement information being greater than a first threshold, the center of gravity displacement information being less than or equal to a second threshold, and the angle change information being less than or equal to a third threshold, determining that the second action is a sliding action; or, In response to the total displacement information being less than or equal to the first threshold, the center of gravity displacement information being greater than the second threshold, and the angle change information being less than or equal to the third threshold, determining that the second action is a rolling action; or, In response to the total displacement information being less than or equal to the first threshold, the center of gravity displacement information being less than or equal to the second threshold, and the angle change information being greater than the third threshold, the second action is determined to be a rotation action.

3. The method according to claim 1, characterized in that The determining, according to the total displacement information, the center of gravity displacement information, and the angle change information, a second action corresponding to the second fingerprint sensor includes: Determining a second action corresponding to the second fingerprint sensor by a pre-trained classifier according to the total displacement information, the center of gravity displacement information and the angle change information; The classifier learns the mapping relationship between the total displacement information, the center of gravity displacement information, the angle change information and the action through a machine learning method.

4. The method according to claim 1, characterized in that The step of determining an input interaction instruction matching the input interaction according to the application scenario and action information of the smart device includes: Based on a plurality of continuous fingerprint frame images corresponding to each fingerprint sensor and pre-registered user fingerprint information, identifying finger information of a touch operation on each fingerprint sensor; Determine a mapping relationship between actions, fingers and input interaction instructions based on the application scenario; An input interaction instruction matching the input interaction is determined according to the action information, the finger information and the mapping relationship.

5. The method according to any one of claims 1 to 4, characterized in that The input interaction instruction includes an action execution instruction and / or a symbol input instruction of the smart device.

6. The method according to claim 1, characterized in that The method further comprises: Interaction feedback information is generated based on the touch operation, where the interaction feedback information is used to provide touch feedback to the user, and the interaction feedback information includes at least one of the following: image feedback information, sound feedback information, and vibration feedback information.

7. The method according to claim 1, characterized in that The at least one fingerprint sensor is located in a side area of ​​the smart device.

8. A fingerprint-based input interaction device, the device being arranged in a smart device, characterized in that: include: an acquisition module, configured to acquire, in response to a user using at least one fingerprint sensor to perform input interaction with the smart device, a plurality of continuous fingerprint frame images corresponding to each fingerprint sensor when the user performs a touch operation on the at least one fingerprint sensor; an identity authentication module, configured to authenticate the user based on a plurality of continuous fingerprint frame images corresponding to one or more fingerprint sensors of the at least one fingerprint sensor and pre-registered user fingerprint information; A first determination module is configured to determine, in response to the user's identity authentication being passed, whether the touch operation corresponding to each fingerprint sensor is a click action type according to change information of a plurality of continuous fingerprint frame images corresponding to each fingerprint sensor in a time series; a second determining module, configured to extract, for a first fingerprint sensor whose touch operation is a click action type among the at least one fingerprint sensor, contact time variation information from a plurality of continuous fingerprint frame images corresponding to the first fingerprint sensor, and determine a first action corresponding to the first fingerprint sensor according to the contact time variation information; a third determination module, configured to extract, for a second fingerprint sensor whose touch operation is a non-click action type among the at least one fingerprint sensor, total displacement information, center of gravity displacement information, and angle change information of fingerprint features in a plurality of continuous fingerprint frame images corresponding to the second fingerprint sensor, and determine a second action corresponding to the second fingerprint sensor according to the total displacement information, the center of gravity displacement information, and the angle change information; The fourth determination module is used to determine an input interaction instruction matching the input interaction according to the application scenario and action information of the smart device, where the action information includes the first action and / or the second action.

9. A smart device, characterized in that: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 7 when executed by a processor.

Images