Using neural and neuromechanical fingerprints to identify changes in a user's health

Abstract

The present application relates to determining a change in a user's health using neural and neural mechanical fingerprints. According to one embodiment, a method for determining a change in a user's health is disclosed. The method includes: sensing multidimensional motion of a body part of a user at a first time and date to generate a first multidimensional motion signal; generating a first neural mechanical fingerprint in response to the first multidimensional motion signal; generating a first health metric in response to the first NFP; sensing multidimensional motion of the body part of the user at another time and date to generate another multidimensional motion signal; generating another neural mechanical fingerprint in response to the another multidimensional motion signal; generating another health metric in response to the another NFP; and comparing the first health metric with the another health metric to determine a difference representing a deterioration in the user's health.

Description

[0001] Information about divisional applications

[0002] This case is a divisional application. The parent case of the divisional application is an invention patent application with application date of February 4, 2016, application number 201680018319.4, and invention name “Using neural and neural mechanical fingerprints to determine health changes of users”.

[0003] CROSS REFERENCE TO RELATED APPLICATIONS

[0004] This Patent Cooperation Treaty (PCT) patent application claims the benefit of U.S. Provisional Patent Application No. 62 / 112,153, filed on February 4, 2015 by inventor Martin Zizi et al., entitled “LOCAL USER AUTHENTICATION WITH NEURO-MECHANICAL FINGERPRINTS,” and Non-Provisional Patent Application No. 15 / 013,764, filed on February 4, 2015 by inventor Martin Zizi et al., entitled “DETERMINING HEALTH CHANGE OF A USER WITH NEURO AND NEURO-MECHANICAL FINGERPRINTS.” Technical Field

[0005] The present embodiment generally relates to health analysis and diagnosis of a user. Background Art

[0006] Wide area network connections via the Internet interconnect many electronic devices (such as computers and mobile smart phones) with remote servers and remote storage devices, so that cloud computer services can be provided. More electronic devices are ready to be interconnected via the Internet as radio transmitters / receivers are added to them.

[0007] Access to some electronic devices and databases by users is usually performed through a login name and password. As more portable electronic devices are used (such as laptops and mobile smart phones), in a highly mobile computing environment, proper authentication of people and devices becomes very important to determine authorized use and reduce the risk of miscommunication of related data. For example, as more mobile health electronic devices are introduced, the privacy of health data captured by mobile health devices becomes very important. As more banking and payments are performed using mobile electronic devices, authorized use becomes very important.

[0008] Authentication of a user using a local electronic device is typically performed by a remote server. Software applications executed by the local electronic device typically save the user's login name and password to make the electronic device and its software applications easier to use. Protecting local access to the user's electronic device has become increasingly important to protect the user and his / her login name and password in the event that the electronic device is lost or stolen. With electronic devices now being used to make payments such as credit card transactions, protecting local access to the user's electronic device has become even more important.

[0009] Reference Figure 1 , various known behavioral recognition methods and anatomical recognition methods are described. The behavioral recognition method is related to the user's behavior or his / her habits. The known anatomical recognition method is related to the user's physical characteristics, such as fingerprints, eye iris scans, veins, facial scans, and DNA.

[0010] Biometrics is used to better authenticate users in order to provide more protection for mobile electronic devices. Biometrics has been pursuing anatomical-based methods (i.e., physical features such as fingerprints, eye iris scans, veins, facial scans, DNA, etc.) and habit (behavioral) methods (typing or keystrokes, handwritten signatures, voice or voice inflection). For example, an anatomical aspect of the user (such as a fingerprint) is used to locally authenticate the user and restrict access to the electronic device. As another example, hand morphology or vein location may be used in conjunction with image analysis to locally authenticate the user and restrict access to the electronic device.

[0011] Behavioral aspects of the user may also be used to better authenticate the user in order to provide more protection for the mobile electronic device. Behavioral aspects relate to a user's unique habits, tastes, and personal profile of behavior. For example, the way a user signs is a unique behavioral aspect that can be used to verify the user's identity.

[0012] However, both the known anatomical and known behavioral aspects of user identification are not foolproof. For example, known user data used for comparison can be stolen from a server and used without the user's knowledge. Additional hardware is typically required to capture biometrics from the user, which results in increased costs. In the case of known added hardware, the risk can increase, where a user can be forced by another user to unwillingly capture biometric data. The behavioral aspects can be slightly invasive and raise privacy issues. Typically, sensing of the behavioral aspects is performed using software, which requires the system to remain on, consuming energy that is typically stored in a rechargeable battery. In addition, the use of behavioral biometrics typically requires the added expense of maintaining a remote centralized database. Summary of the invention

[0013] Using biometrics with known anatomical and / or known behavioral aspects of the user for identification has some disadvantages.

[0014] What is needed is a user authentication solution that is easy to use, decentralized so it can be used locally, is substantially fail-safe against predators, consumes low power for battery applications, and respects the privacy of users by protecting their data so that it can be easily adopted.

[0015] The user authentication solution disclosed herein provides fail-safe authentication in a decentralized mobile environment while protecting the user's privacy and data.The user authentication solution disclosed herein employs different forms of tamper-resistant biometrics.

[0016] The user authentication solution disclosed herein may employ 3D sensors and signal processing to capture signals representing the user's neuromuscular microfunctions that are translated into well-defined micromotions ("micromotion signals"). Signal processing algorithms and feature extraction are then used to capture unique signal features in the micromotion signals. These unique signal features associated with the user's neuromuscular micromotions may be used to uniquely identify the user, somewhat similar to a fingerprint. Therefore, these unique signal features associated with the user's neuromuscular micromotions are referred to herein as "neuromechanical fingerprints" (NFPs). Neuromechanical fingerprints may be used for local user authentication at mobile electronic devices (such as laptops or smart phones) or at other types of electronic devices. Using the user's NFP captured by the sensor and extracted by the algorithm avoids profiling and storing the user's habitual or patterned behaviors.

[0017] Embodiments of a user authentication solution that employs sensors and signal processing algorithms to generate and reproduce a neuromechanical fingerprint (NFP) of an authorized user are disclosed through the figures and detailed description herein.

[0018] 1. A method for determining a change in a user's health, the method comprising: a) sensing multi-dimensional motion of a body part of a user at a first time and date to generate a first multi-dimensional motion signal; b) generating a first neural mechanical fingerprint (NFP) in response to the first multi-dimensional motion signal; c) generating a first health metric in response to the first NFP; d) sensing multi-dimensional motion of the body part of the user at another time and date to generate another multi-dimensional motion signal; e) generating another neural mechanical fingerprint in response to the another multi-dimensional motion signal; f) generating another health metric in response to the another NFP; and g) comparing the first health metric with the another health metric to determine a difference representing a change in the user's health. 2. The method according to technical solution 1, further comprising: storing the first health metric associated with the first time and date; and storing the another health metric associated with the another time and date. 3. The method according to technical solution 1, further comprising: repeating steps d to g to continue determining the difference representing the change in the user's health. 4. The method according to technical solution 3, wherein the difference between the first health metric and the another health metric is positive, which indicates health improvement. 5. The method according to technical solution 3, wherein the difference between the first health metric and the other health metric is negative, which indicates health degradation. 6. The method according to technical solution 5, wherein the health degradation is of a type associated with drinking; and the health degradation of the user limits the user's ability to operate a machine or vehicle. 7. The method according to technical solution 1, wherein the health degradation is the user's heart rate variability. 8. The method according to technical solution 5, wherein the health degradation is a neurodegenerative disease of the user. 9. The method according to technical solution 8, wherein the neurodegenerative disease is Alzheimer's disease or Parkinson's disease. 10. The method according to technical solution 1, wherein the multi-dimensional motion of the body part of the user is locally sensed at a first time and date to generate the first multi-dimensional motion signal; the first multi-dimensional motion signal is remotely transmitted to a server via a communication system to store the first multi-dimensional motion signal and generate a first neuromechanical fingerprint (NFP) in response thereto. 11. The method according to technical solution 1, wherein the multi-dimensional motion of the body part of the user is three-dimensionally sensed using a three-dimensional touch sensor at the first time and date to generate a three-dimensional motion signal. 12. A method according to technical solution 1, wherein at the first time and date, a three-dimensional accelerometer is used to three-dimensionally sense the multi-dimensional movement of the body part of the user to generate a three-dimensional motion signal.

[0019] 13. A method for determining a change in a user's health, the method comprising: sensing multidimensional motion of a body part using a multidimensional sensor at multiple dates / times; generating multiple multidimensional motion signals at the multiple dates / times in response to the sensing; generating multiple neural mechanical fingerprints for the multiple corresponding dates / times in response to the multiple multidimensional motion signals; comparing at least two of the multiple neural mechanical fingerprints to determine a change in the user's health between at least two dates / times; and reporting the health change to the user. 14. The method according to technical solution 13, wherein the comparison analyzes the multiple neural mechanical fingerprints that change over time, and the method further comprises: analyzing a frequency component in each of the multiple multidimensional motion signals to determine a pathological state. 15. The method according to technical solution 13, further comprising: searching for one or more predetermined data patterns in each of the multiple neural mechanical fingerprints to determine a pathological state. 16. The method according to technical solution 14, further comprising: searching for one or more predetermined data patterns in each of the multiple neural mechanical fingerprints to determine another pathological state. 17. The method according to technical solution 13, further comprising: storing the plurality of multi-dimensional motion signals in a storage device for subsequent processing. 18. The method according to technical solution 17, further comprising: searching for one or more predetermined data patterns in each of the plurality of multi-dimensional motion signals to determine a pathological state.

[0020] 19. A system for analyzing a user's health, the system comprising: a health diagnostic device, comprising: a multidimensional sensor for sampling multidimensional motion of a user's body part and generating a multidimensional motion signal in response thereto; and a controller coupled to the multidimensional sensor, the controller for controlling the sampling of the multidimensional motion and the generation of the multidimensional motion signal at different dates and times; and a server in communication with the health diagnostic device, the server comprising: a storage device for storing data and providing access to the stored data; and one or more processors coupled to the storage device, the one or more processors executing instructions to configure the one or more processors to process data, including data stored in the storage device, wherein the storage device stores multidimensional motion signals and corresponding neural mechanical fingerprints at different dates and times for processing by the one or more processors, wherein at least one processor is configured by instructions to generate a plurality of neural mechanical fingerprints for the plurality of corresponding dates / times in response to the plurality of multidimensional motion signals; and wherein at least one processor is configured by instructions to compare at least two of the plurality of neural mechanical fingerprints to determine a change in the user's health between at least two dates / times. 20. The system according to technical solution 19, wherein at least one processor is configured by instructions to analyze the frequency components in each of the multi-dimensional motion signals to determine another pathological state. 21. The system according to technical solution 19, wherein at least one processor is configured by instructions to search for one or more predetermined data patterns in each of the plurality of neural mechanical fingerprints to determine a pathological state. 22. The system according to technical solution 20, wherein at least one processor is configured by instructions to search for one or more predetermined data patterns in each of the plurality of neural mechanical fingerprints to determine another pathological state. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 It is a background diagram illustrating various behavioral and physiological recognition methods.

[0022] Figures 2A to 2D An example of a Poincare phase scatter plot showing gravity-corrected three-dimensional accelerometer data captured from different users is illustrated.

[0023] Figure 3A This is a diagram for explaining remote authentication of a user via the Internet using a server and a storage device (cloud storage device) in a storage area network.

[0024] Figure 3B is a diagram illustrating local authentication of a user at a local electronic device.

[0025] Figure 4is a diagram illustrating authentication techniques that may be used in conjunction with NFP authentication to provide multi-factor authentication.

[0026] Figure 5 This is a chart comparing various biometric identification technologies with NFP authentication.

[0027] Figure 6 is a table of tremor types and associated frequencies and conditions.

[0028] Figure 7 is a functional block diagram of an example of an electronic device including a sensor for capturing micro-motion signals from a user and an NFP authentication system for controlling access in response to the micro-motion signals.

[0029] Figures 8A to 8C It is an NFP authentication system with different sensors used to sense micro-movements from the fingers or from the hand.

[0030] Fig. 9 A waveform graph of the acceleration measured at the user's hand to illustrate the difference between large and small movements.

[0031] Fig. 10A A diagram illustrating device orientation and world orientation.

[0032] Fig. 10B is a diagram illustrating the use of eigenvectors to transform 3D data samples in a data set from a device orientation to a world orientation.

[0033] Fig.11A It is a functional block diagram of a bandpass filter used to filter out signals outside the frequency range of desired vibration to capture micro-motion signals.

[0034] Fig. 11B It is used to achieve Fig.11A The functional block diagram of the high pass filter and low pass filter has the same result as the band pass filter.

[0035] Fig. 12A This is a functional block diagram of a gravity high-pass filter used to filter out the effects of gravity.

[0036] Fig. 12B is a diagram illustrating coordinate transformation of data samples in a data set to move the center of gravity to the origin of the three axes in order to eliminate the influence of gravity.

[0037] Fig.13A is a waveform diagram of a raw, unfiltered sample set of a three-dimensional acceleration data signal from a 3D accelerometer.

[0038] Fig. 13B Is the 3D micro motion signal in Fig.13AThe waveform of the original signal waveform shown in the figure is after preprocessing, bandpass filtering and gravity compensation.

[0039] Fig.14A It is the functional block diagram of the NFP authentication controller.

[0040] Fig. 14B yes Fig.14A Functional block diagram of the NFP certification classifier shown in .

[0041] Fig.15 Is Fig.14A Diagram of access and recalibration control performed by the authentication controller shown in FIG.

[0042] Fig.16 is a plot of the power spectral density of micro-movement signals associated with tremor.

[0043] Fig.17A It is through Fig. 13B Plot of the CEPSTRUM waveform resulting from performing CEPSTRUM analysis on the micromotion waveform shown in FIG.

[0044] Fig. 17B yes Fig.17A An enlarged view of a portion of a plot of the CEPSTRUM waveform shown in FIG.

[0045] Figures 18A to 18B An example of different NFPs for a single axis based on the frequencies of two different users is illustrated.

[0046] Figures 19A to 19B An example of different NFPs for a single axis based on time for two different users is illustrated.

[0047] Fig. 20 A hidden Markov model that can be used as an NFP classifier model to determine a match percentage from the NFP and authorized user calibration parameters is described.

[0048] Fig.21 A block diagram illustrating a process for analyzing a user's health using neural and neuromechanical fingerprints. DETAILED DESCRIPTION

[0049] In the following detailed description of the embodiments, numerous specific details are set forth to provide a thorough understanding. However, it will be apparent to those skilled in the art that the embodiments can be practiced without these specific details. In other instances, well-known methods, procedures, components, and circuits have not been described in detail to avoid unnecessarily obscuring aspects of the embodiments.

[0050] Embodiments include methods, devices, and systems for forming and utilizing a neuromechanical fingerprint (NFP) to identify and authenticate a user.

[0051] Introduction

[0052] Certain user movements are habitual or part of the user's motion repertoire. A user signing a document is, for example, a situational movement of the user that is a result of behavioral habit. Signature movements that are usually analyzed are large or large movements made by the user with a writing instrument. For example, from the large movements of the signature, it can be determined, for example using the eyes, whether the writer is left-handed or right-handed.

[0053] While these large movements can be useful, there are also micro-movements (extremely small movements) that users make when signing, doing other movements, or just resting without exercising. These micro-movements are neurally derived or neural-based and invisible to the eye. These micro-movements of the user are attributed to each person's unique neuromuscular anatomy and may also be referred to herein as neurally derived micro-movements. These micro-movements are also related to the motor control process from a person's motor cortex to his / her hands. Using one or more sensors, signal processing algorithms, and / or filters, electronic signals ("motion signals" and "micro-motion signals") containing the user's neurally derived micro-movements can be captured. Of particular interest are micro-motion electronic signals that represent the user's micro-movements within the motion signal.

[0054] The user's micro-movements are related to the cortical and subcortical control of motor activity in the brain or elsewhere in the human body's nervous system. Like a mechanical filter, a person's specific musculoskeletal anatomy can affect the user's micro-movements and contribute to the motion signal containing the user's micro-movements. The motion signal captured from the user may also reflect part of the proprioceptive control loop containing the brain and proprioceptors present in the user's body.

[0055] When the motion signals are properly analyzed for micro-motion signals representing the user's micro-motions, the resulting data can generate unique and stable physiological identifiers, more specifically neural identifiers, that can be used as unwritten signatures. These unique identifiers are the user's neuromechanical fingerprints. Neuromechanical fingerprints may also be referred to herein as NeuroFingerPrint (NFP).

[0056] The user's motion can be captured by various electronic sensors, including the user's large motion and the user's micro-motion. For example, an electronic touch sensor or an electronic accelerometer can be used to capture the user's large motion and micro-motion. U.S. Patent Application No. 13 / 823,107, filed by Geoff Klein on January 5, 2012 and incorporated herein by reference, describes how an accelerometer can be used to quietly capture the user's time-varying motion data when the user operates an electronic device and generate a motion spectrum that can be used to identify the user. U.S. Patent Application No. 13 / 823,107 utilizes a spectrum or motion habits. The embodiments disclosed herein neither establish a motion spectrum nor rely on a motion spectrum. The embodiments disclosed herein extract signals related to the quality control mechanism of the nervous system. U.S. Patent Application No. 13 / 823,107 ignores the neural-derived micro-motion of the user that the embodiments disclosed herein are interested in. In addition, in generating a neural mechanical fingerprint, the large motion signal is suppressed or filtered to capture the micro-motion signal component. When using a three-dimensional accelerometer to capture the user's micro-motion, the large motion signal needs to be filtered out from the micro-motion signal data. The vectors due to gravity are also ignored in U.S. Patent Application No. 13 / 823,107. When using a three-dimensional accelerometer, gravity is compensated or filtered out in generating micro-motion signals.

[0057] Reference Figures 2A to 2D , shows examples of three-dimensional Poincare phase scatter plots for four different users. Each three-dimensional Poincare phase scatter plot shows a pattern 200A-200D of gravity-corrected three-dimensional accelerometer data.

[0058] Raw accelerometer data was obtained using the same electronic device over the same time period with a user making large movements with their hand. The electronic device had a three-dimensional accelerometer sensor for capturing raw accelerometer data in three dimensions X, Y, and Z.

[0059] Signal processing is performed on the raw accelerometer data to filter or suppress unwanted signals, correct for gravity, and extract signals representing neurally derived micro-movements of the user's hand or fingers (micro-motion signals). The three-dimensional x(t), y(t), z(t) of the micro-motion signals may be further processed into phases x(t), y(t), z(t) during a sampling time period for the user and plotted as a three-dimensional Poincare phase scatter plot. If the sampling time period is shorter, the micro-motion signals x(t), y(t), z(t) sensed from the samples of the user's neuromuscular micro-movements may be stored in digital form in a storage device to represent a neuromechanical fingerprint (NFP). The digital form of the micro-motion signals may be encrypted using an encryption algorithm and an encryption key prior to storage in order to protect them from unauthorized access by unauthorized personnel.

[0060] If available Figures 2A to 2C As can be readily seen in FIG. 1 , the patterns 200A-200D in each of the Poincare phase scatter plots generated from the user's neural-derived motion are substantially different. For example, the centroid 202A-202D of each pattern 200A-200D is different. Other characteristics of each pattern 200A-200D are also different for each user. Thus, the pattern of neural-derived motion is unique to each user and can be used to uniquely identify the user.

[0061] The unique patterns 200A to 200D and the NFP in the generated Poincare phase plot are generally stable. Therefore, typically whenever a user touches or moves the sensor, the unique patterns 200A to 200D and the NFP can be repeatedly sensed within a sampling time period and then compared with the initially calibrated NFP using an algorithm to authenticate the user's identity. However, if there is a progressive neuromuscular disease, the user's unique patterns 200A to 200D are less stable. If this is the case, the neural algorithm can be periodically recalibrated to compensate for disease progression. For example, an elderly person with a neuromuscular disease can easily recalibrate the neural algorithm weekly depending on the drift cutoff regarding the initial calibration of the NFP. As another example, a user who is being treated with a motion-changing drug may also periodically recalibrate the NFP neural algorithm. For most users, it will be rare to recalibrate the NFP neural algorithm after the initial calibration.

[0062] The analysis of the motion signal of the micro-movement uses a neural algorithm to obtain the user's neural mechanical fingerprint (NFP). Therefore, this neural algorithm may be referred to as an NFP neural algorithm in this article.

[0063] NFP Neural Algorithm

[0064] The NFP Neural Algorithm is not an algorithm that uses neural network training, classifiers, or deep learning. The NFP Neural Algorithm is an algorithm that specifically collects, isolates, and analyzes signals from the human nervous system and its interactions with parts of the body connected to the nervous system, such as the muscle system, glandular cells, and / or skin. Activities of interest for analyzing micro-motion signals include motor control originating from our nervous system, sensory input, and stress responses that are unique to each person's anatomy and nervous system.

[0065] Consider, for example, a user moving his or her hand. For example, a motion algorithm may record the physical movement over time and analyze its speed, distance, direction, and force. A neural algorithm collects, isolates, and analyzes the neural activity related to the hand movement over time from electronic signals (e.g., motion signals including micromotion signals) without performing a brain scan using a brain scanner.

[0066] While kinematic movement of a body part may be analyzed for micro-motion using an electronic accelerometer, the touch of one or more fingers to the touch sensor may be analyzed, for example, using an electronic touch sensor (e.g., a touch pad). The touch sensor may be used to generate a micro-motion signal representing micro-motions that vary over time as the fingers touch the touch sensor. The touch sensor may generate a three-dimensional signal representing the X and Y position of the finger relative to the touch sensor and the Z position representing the pressure applied to the touch sensor. The micro-motions may be included as part of each of the three-dimensional position signals. Using a touch sensor, gravity is generally not a factor that needs to be corrected.

[0067] By focusing on micro-motion signals rather than large-motion signals, touch sensors can be used in conjunction with neural algorithms to better mimic human cognitive interfaces in machines. This can improve human-machine interfaces. For example, consider the human cognitive interface between a husband and wife or between close people. When a husband touches his wife's arm, the wife can often recognize that it is her husband who is touching her just from the feeling of that touch because she is familiar with his touch. If the touch feeling is unique, then a person can usually recognize what is touching him / her just from that unique feeling.

[0068] Touch recognition

[0069] The technology of using touch sensors to sense and capture micro-motions and using neural algorithms to analyze NFPs may be referred to herein as touch recognition. A system that includes touch recognition for authenticating and / or identifying a user may be referred to herein as a touch recognition system. Figure 1 , touch recognition using NeuroFingerPrint is a physiological user identifier. Touch recognition is not an anatomical user identifier. Touch recognition is not a behavioral user identifier. In contrast to anatomical recognition methods, which are fixed attributes of the user, physiological recognition methods are related to the user's functional body. Behavioral recognition methods are related to the user's movements or his / her habits.

[0070] There are inherent advantages to using touch sensors and neural algorithms in user authentication. The user interface for user authentication can be intuitive and related to the spontaneous behavior of human users with only minimal training, if no training is required at all. The user experience of the user authentication process can be smooth. The user does not need to perform any specific user authentication tasks other than those typically associated with the use of the user electronic device. For example, after calibration, the user only needs to hold his / her smartphone, such as to make a call, so that the accelerometer and neural algorithm perform the user authentication process.

[0071] The NeuroFingerPrint may be captured non-intuitively by the touch sensors and NFP neural algorithm. The NeuroFingerPrint may be isolated from other user motions. Thus, one or more touch sensors used to capture keystrokes or presses may also be used to capture micro-motions of the user and generate micro-motion signals and NeuroFingerPrints in response thereto. For example, a user may simply select one or more buttons (e.g., dial a phone number to make a call, enter a personal identification number, select a function button or a power button) to cause the underlying one or more touch sensors and neural algorithms to perform a user authentication process.

[0072] For example, in the case where there is no change in the user experience, the NFP may be generated by the NeuroFingerPrint (NFP) neural algorithm while typing in the user's personal identification number (PIN). Alternatively, for example, the NFP may be generated by the NeuroFingerPrint (NFP) neural algorithm while pressing a power button or a function button (e.g., a home button). In this case, the NeuroFingerPrint (NFP) may be used for user authentication without a PIN number. Alternatively, in the case where there are few or no changes to the user interface, the NeuroFingerPrint (NFP) may be used in conjunction with a PIN number for user authentication.

[0073] With little or no change to the user interface, one or more touch sensors associated with a keypad or button may also be used to capture micro-motions and generate micro-motion signals as a user presses or selects a key of the keypad or button. A keypad or button supporting NFP capture may be used to control entry and / or exit from an office, building, or other real estate, such as a residence, business, government office, or other secure facility.

[0074] Furthermore, a touch sensor combined with a neural algorithm may advantageously be built into a physical electronic key, which may then be used wirelessly to open a door, start a car engine, or gain access to a server or other information technology system.

[0075] Local and remote user authentication

[0076] Reference Figures 3A to 3B . Figure 3A Describes remote authentication of users. Figure 3B Describes local authentication of users.

[0077] Touch recognition facilitates local authentication of the user at the local electronic device, e.g. Figure 3B. NFP user authentication does not require the use of a large remote database. The NFP user authentication system can be self-contained within the local electronic device, such as a mobile smart phone. As remote authentication may require: the data used for user authentication does not need to be stored in a storage device on a storage area network ("cloud") and does not need to be accessed from the storage device via the Internet. As remote authentication may require: the login ID and password used for user authentication does not need to be sent to a server via the Internet to authenticate the user. Touch recognition is decentralized and can be self-contained within the local electronic device.

[0078] In the case of remotely authenticated users, e.g. Figure 3A As shown in , users are open to the Internet and can be subject to the risks associated with it. In the case of local authentication, e.g. Figure 3B As shown in , a user can communicate with, for example, Figure 3B Alternatively, a token representing local user authentication may be generated by a local electronic device and sent from the local electronic device to a local server by wire and / or wirelessly or remotely to a remote server via an Internet cloud of routers, switches, and communication systems, such as Figure 3B As shown in .

[0079] While touch recognition facilitates local authentication, touch recognition can be easily combined with a remote authentication technique or another local authentication technique to provide an additional level of security.

[0080] Multi-factor authentication and NeuroFingerPrint

[0081] Reference Figure 4 , user authentication systems may employ multiple factors to authenticate a user. Multi-factor authentication combines two or more authentication techniques. Previously, multi-factor authentication was not user-friendly and was therefore avoided. User-friendly authentication methods are instead usually preferred, but result in compromised protection.

[0082] As mentioned herein, touch recognition or recognition using an accelerometer may be combined with a remote authentication technique or another local authentication technique to make multi-factor authentication more user-friendly.

[0083] Touch recognition can be combined with a password, a personal identification number, a pattern, or something else that the user knows or remembers. Alternatively, touch recognition can be combined with a token, a smart card, a mobile token, an OTP token, or some other user authentication device that the user has. Alternatively, touch recognition can be combined with anatomical or behavioral user biometrics. For example, neural-based touch recognition can be easily combined with fingerprint recognition to provide a user-friendly multi-factor authentication system.

[0084] The banking industry is particularly interested in avoiding additional burdens on users when user authentication is required. In addition, the banking industry is interested in high-performance user authentication systems that provide reliable user authentication and are difficult to circumvent. Neural-based touch recognition can be easily combined with other authentication technologies that impose minimal burden on users. In addition, the neural-based touch recognition disclosed herein can be substantially reliable and extremely difficult to circumvent.

[0085] Comparison of Biometric Identification Technologies

[0086] Reference Figure 5 , a chart illustrating a comparison matrix of various biometric identification technologies. A relative rating is used to compare user authentication using NFP with other biometric user identification technologies. The relative rating is based on seven grading criteria. Each criterion can be graded as high (H), medium (M) or low (L) for each biometric identification technology. From left to right, the seven grading criteria are persistence, collectability, performance, acceptability, avoidance, uniqueness and versatility. NFP is rated higher than other biometric identification technologies for the following reasons.

[0087] NFP is a high ranking universal technique. Each person of the user will have a unique neural system from which NFP can be captured.

[0088] NFP is a highly unique user authentication technology that distinguishes users. NFP reflects the user's motor control of the user's body. Motor control is related to brain activity that controls muscle tension in the hands. Even twins will have different motor control because they will have experienced different life experiences and have different motor skill training. For example, twins will learn to ride a bicycle differently, thereby learning different motor control of muscles that will be reflected in different NFPs.

[0089] NFP is a relatively highly persistent or stable technology because it is related to each person's anatomy. Evolutionary processes (such as neurological diseases) that can affect the stability of NFP are rare. Even in those rare cases, the neural evolutionary process is usually slow enough so that the NFP user authentication system can be recalibrated. The NFP used for touch recognition is different for the user's left hand and the user's right hand. However, the NFP is a specific NFP for each user. The user can be consistent with which hand / which finger he / she uses to hold / touch the accelerometer-enabled device / touch sensor-enabled device. Alternatively, the NFP user authentication system can be used to calibrate multiple NFPs for the same user. In this case, the touch of any one hand or multiple fingers can authenticate the user of the electronic device.

[0090] Some neurotropic drugs may alter the scanned NFP. However, the NFP user authentication system may be recalibrated to compensate. In other cases, changes in the scanned NFP due to the ingestion of alcohol, drugs, or strong medications may be used to improve vehicle safety.

[0091] Alcohol can temporarily change the scanned NFP so that it does not match the calibrated NFP. This can be advantageously used to prohibit driving a car or other vehicle under the influence of alcohol. For example, NFP touch technology can be implemented in a car key or car start button. If the scanned NFP is not affected by alcohol, then the NFP touch technology allows the key or start button to start the engine to drive the car. If the scanned NFP is affected by the user's drinking, then the NFP touch technology can prohibit the key or start button from starting the car engine.

[0092] NFP is relatively easy to collect or capture because accelerometers have become common in mobile devices. Accelerometer values ​​are frequently sampled using a clock every microsecond or about every microsecond to capture micro-movements of the user. In addition, accelerometers are relatively sensitive. Accelerometers can typically sense sub-micron accelerations so that, for example, they can capture micro-movements of a user's hand. Various three-dimensional accelerometers can be used to capture micro-movements of a user's hand. Various touch sensors can be used to capture micro-movements and types (whether capacitive or resistive is irrelevant).

[0093] The performance of NFP authentication using NFP is high. Using a non-optimal mathematical algorithm as a neural algorithm can achieve a recognition success rate between 93% and 97%. A more appropriate and accurate mathematical algorithm that takes into account the variables of time and trajectory can achieve a higher recognition success rate to reach 100% recognition of the correct user. In the event that the NFP authentication system cannot identify the correct user, an alternative user authentication system, such as a PIN, can be used in parallel with the NFP authentication. In contrast, most biometrics (except real fingerprints or iris scans) have low performance with a higher failure rate, which can range between 18% and 20%, providing a successful recognition rate between 80% and 82%.

[0094] NFP technology should have a high degree of acceptability because it is a user-friendly authentication method. NFP user authentication systems can be quite transparent or smooth. However, acceptability is also governed by how the NFP user authentication system is designed and sold, as well as how it is supported by appropriate security measures and independent audits.

[0095] NFP is difficult to circumvent, so it has a high circumvention factor rating. When the NFP authentication system resides in an electronic device and is not available to the Internet, it is almost impossible to gain access through hacking, spoofing, or reverse engineering.

[0096] Initially, a calibration NFP file (user calibration parameters) is electronically generated locally for the user by the NFP user authentication system. The calibration NFP file is typically encrypted and stored locally on the electronic device (e.g., in a memory associated with the NFP authentication controller) and is therefore not available on the Internet. However, when stored elsewhere (e.g., a storage device in a storage area network or a storage device associated with an authentication server), the calibration NFP file should be encrypted to provide greater security.

[0097] NFP user authentication does more than just perform a file comparison to grant access. NFP must be regenerated from the user's body in real time to gain access. This prevents hackers from using stolen files to gain access and authorization to electronic devices by using surrogate misleading inputs or electronic spoofing or phishing for information. In order to circumvent service-level NFP user authentication, hackers will have to perform data acquisition simultaneously and in parallel with the authorized user while the user is accessing a service that requires authentication or login. In order to circumvent device-level NFP user authentication, hackers will have to mimic the neural control of the authorized user to obtain NFP data and then type it into the device to gain access. Hackers can steal calibrated NFP values ​​from the authorized user's device, but then the hacker will have to incredibly regenerate the user's micro-motion input into a second electronic device using the same calibrated NFP authentication system. Therefore, electronic devices with NFP authentication systems can be extremely difficult to circumvent. Mobile electronic devices with NFP authentication systems can ultimately be trusted as gatekeepers for users who provide real-time verification identification.

[0098] Only when the user utilizes the electronic device, a scanned NFP is then generated in real time, wherein the calibration NFP only adjusts the calculation. In the absence of a calibration NFP file generated by the correct user, the NFP authentication system denies local access to the electronic device. If the correct user is not alive, the NFP authentication system will deny access because the micro-motion signals of the correct user cannot be generated. Therefore, circumvention of the NFP authentication technology is difficult. In contrast, classic fingerprints have a high level of difficulty to circumvent, however, even classic fingerprints can be stolen and can be reverse engineered to log into the fingerprint authentication system.

[0099] In summary, NFP technology, such as biometrics used in user authentication systems, ranks high compared to other biometric technologies.

[0100] Privacy Protection

[0101] Mobile electronic devices are being used in conjunction with health and fitness applications. In some cases, mobile electronic devices can be used as a user interface to connected medical devices. Rules and laws in several countries regulate the privacy of users' medical conditions. Therefore, the protection of users' medical conditions and medical data has become increasingly important.

[0102] The NFP certification system may be used to help protect user data that may be stored or entered into an electronic device. The NFP certification system may be used to help comply with medical data laws and regulations in effect in several countries.

[0103] NFP data generated from micro-movements due to the user's nervous system may be considered a form of medical data.The NFP authentication system may be implemented to effectively protect calibrated NFP data stored in an electronic device or elsewhere when used for authentication purposes.

[0104] Micromovements and tremors

[0105] NFP is produced in response to micro-movements associated with a type or form of tremor. Tremors are involuntary, rhythmic muscle movements that cause oscillations in one or more parts of the human body. Tremors may or may not be visible to the naked eye. Visible tremors are more common in middle-aged and elderly people. Visible tremors are sometimes seen as a disorder in the part of the brain that controls one or more muscles throughout the body or a specific area, such as the hands and / or fingers.

[0106] Most tremors occur in the hands. Therefore, tremors with micro-movements can be sensed when holding a device with an accelerometer or by touching a touch pad sensor with a finger.

[0107] There are different types of tremors. The most common form or type of tremor occurs in healthy individuals. Normally, healthy individuals do not notice this type of tremor because the movements are too small and can occur while performing other movements. The micro-movements of interest associated with one type of tremor are so small that they are not visible to the naked eye.

[0108] Tremors can be activated under various conditions (resting, postural, kinematic) and can generally be classified as resting tremor, kinetic tremor, postural tremor or kinetic or intention tremor. Resting tremor is a tremor that occurs when the affected body part is not active but against gravity. Kinetic tremor is a tremor due to voluntary muscle activation and includes numerous tremor types, including postural tremor, kinetic or intention tremor and task-specific tremor. Postural tremor is related to making a body part against gravity (such as extending an arm away from the body). Kinetic or intention tremor is related to goal-directed movement and non-goal-directed movement. An example of kinetic tremor is the movement of moving a finger to a person's nose, which is often used to detect drivers driving under the influence of alcohol. Another example of kinetic tremor is the movement of lifting a glass of water from a table. Task-specific tremor occurs during very specific movements, such as when writing on paper using a pen or pencil.

[0109] Tremor (whether visible to the eye or not) is thought to originate from some oscillating neuronal pool within the nervous system, some brain structure, some sensory reflex mechanism, and / or some neuromechanical coupling and resonance.

[0110] Although many tremors have been described as physiological (without any disease) or pathological, it is believed that the tremor amplitude is not very useful for its classification. However, the tremor frequency is of interest. The tremor frequency allows it to be used in a useful way to extract the signal of interest and generate a unique NFP for each user.

[0111] Figure 6 Table illustrating the various tremor types, their activation conditions, and the expected movement frequencies attributed to a given tremor type.

[0112] Many pathological conditions, such as Parkinson's disease (3 to 7 Hz), cerebellar disease (3 to 5 Hz), dystonia (4 to 7 Hz), various neurological diseases (4 to 7 Hz), cause motion / signals to be lower frequencies, such as 7 Hz and below. Since pathological conditions are not common to all users, motion / signals at these frequencies are not conducive to generating NFP and need to be filtered out. However, some of the embodiments disclosed herein are used to specifically focus on those pathological signals as a way to record, monitor, and follow the conditions to determine whether the health condition is sound or degraded.

[0113] Other tremors (e.g., physiological, essential, orthostatic, and augmented physiological tremors) may occur under normal health conditions. These tremors themselves are not diseases. Therefore, the tremors are generally present in the general population. Physiological tremors and other tremors common to all users are of interest because they produce micro-movements at frequencies ranging between 3 to 30 Hz or 4 to 30 Hz. The tremors may be activated when muscles are used to support a body part against gravity. Therefore, holding an electronic device in a person's hand to support the hand and arm against gravity may produce physiological tremors that may be sensed by an accelerometer. Using the fingers of a hand to touch the touch pad of an electronic device and supporting it against gravity may produce physiological tremors that may be easily sensed by a finger touch pad sensor.

[0114] Some tremors may be the result of degenerative conditions or other common conditions that affect the user's health. Given that nerve cells are highly sensitive to metabolism, other conditions such as diabetes, hypertension, heart failure, bacterial or viral infections, and / or parasitic diseases may affect the user's micro-movements (which appear as common data patterns or signatures of neuromechanical fingerprints) or micro-movement signals.

[0115] Essential tremor of the kinetic type may occur and be sensed when a user has to enter a PIN or login ID to gain access to a device or phone. The frequency range of essential tremor is between 4 to 12 Hz, which may be reduced to a frequency range of 8 to 12 Hz to avoid sensing of tremors due to uncommon pathological conditions.

[0116] For physiological tremors (or enhanced physiological tremors, with larger amplitudes as above), the coherence of different body sides is low. That is, the physiological tremors on the left body side are extremely irrelevant to the physiological tremors on the right body side. Therefore, it is expected that the tremors in the left hand or left fingers of the user will be different from the tremors in the right hand or right fingers. Therefore, the NFP authentication system will require the user to be consistent when using the same side of the hand or finger for authentication; or alternatively, multiple authorized user calibration parameter sets (one authorized user calibration parameter set for each hand or one authorized user calibration parameter set for each finger) will be used to extract the NFP.

[0117] Motions with higher frequencies of interest can be considered as noise. Therefore, it is necessary to filter out signals in the raw motion signal that have frequencies above the maximum value in the desired range (e.g., 12 Hz or 30 Hz). Therefore, the frequency signal range from 8 Hz to 12 Hz and / or 8 Hz to 30 Hz contains useful information about micro-motions that can be used to generate NFP.

[0118] The raw signal captured by a finger touch pad sensor in an electronic device or by an accelerometer of a handheld electronic device may have several unwanted signal frequencies therein. Therefore, a type of filtering responsive to filtering out signals outside a desired frequency range may be used to obtain a micro-motion signal from the raw electronic signal. Alternatively, an isolation / extraction means for signals in a desired frequency range may be used to obtain a micro-motion signal from the raw electronic signal. For example, a finite impulse response bandpass filter (e.g., a bandpass of 8 to 30 Hz) may be used to select a low signal frequency range of interest in the raw electronic signal sensed by the touch pad or accelerometer. Alternatively, a low pass filter (e.g., 30 Hz cutoff) and a high pass filter (e.g., 8 Hz cutoff) or a high pass filter (e.g., 8 Hz cutoff) and a low pass filter (e.g., 30 Hz cutoff) may be combined in series to achieve a similar result.

[0119] Electronic device with NFP certification

[0120] Reference Figure 7 , illustrating a functional block diagram of an electronic device 700 with NFP authentication. The electronic device 700 may be, for example, a smart cell phone or other handheld type portable or mobile electronic device that requires access control. The electronic device 700 includes one or more three-dimensional (3D) sensors that can be used to capture raw 3D electronic signals including raw 3D micro-motion signals that can be used for NFP authentication.

[0121] The electronic device 700 may include the following devices coupled together: a processor 701, a storage device (SD) 702, a power source 703 (e.g., a rechargeable battery), a button / pad 704, a keypad 705, a three-dimensional (3D) accelerometer 706 (optionally), a display device 707, one or more radios 708, and one or more antennas 709. If the electronic device does not have a three-dimensional (3D) accelerometer 706, one or more of the button / pad 704, the keypad 705, and the display device 707 are touch-sensitive, so they can be used to capture 3D raw electronic signals including 3D micro-motion signals.

[0122] The storage device 702 is preferably a non-volatile type of storage device that stores data, instructions and (possibly) other information (e.g., user calibration parameters) in a non-volatile manner so that it is not lost when the electronic device enters a sleep state to save power or is completely turned off. The storage device 702 stores software application instructions 712, NFP instructions and (possibly) NFP data 714 (e.g., calibration NFP files of authorized user calibration parameters) for the NFP authentication system and user scribble pad data 716 for the user. The NFP instructions and NFP data 714 are separated from the user scribble pad data 716 and the software application instructions 712 for security reasons so that they are not accessible to all users. The storage device 702 is coupled to the processor 701 so that the data and instructions can be read by the processor and executed to perform the functions of the software application. The NFP instructions and NFP data 714 (if any) are read by the processor to execute the NFP authentication controller module and perform the functions required to provide NFP authentication.

[0123] One or more radios 708 are coupled to the processor 701 and one or more antennas 709 and are coupled between the processor 701 and the one or more antennas 709. One or more radios 708 can receive and transmit data wirelessly via a wireless network. One or more radios 708 can include a Wi-Fi radio for a local wireless (Wi-Fi) network, a cellular radio for a cellular phone network, and a Bluetooth radio for a Bluetooth wireless connection. Software applications that require authentication of a remote server can be executed by the electronic device. The NFP authentication system is used to grant access to the electronic device itself. However, the NFP authentication system can also be used by software applications to verify or authenticate the identity of an authorized user.

[0124] To recharge the battery or provide an alternative power source, a power connector and / or a combined power / communication connector (e.g., a universal serial bus connector) 722 may be included as part of the electronic device 700. The connector 722 may be coupled to the processor 701 for data communications and to the power source 703 to recharge the battery and / or provide an alternative power source to the electronic device 700. For wired connections, the electronic device 700 may further include a network interface controller and connector 724, such as an Ethernet controller and port connector (e.g., RJ-45), coupled to the processor.

[0125] Most portable or mobile electronic devices now include a 3D accelerometer 706. Traditionally, the 3D accelerometer 706 has been used to determine the orientation of the electronic device. However, the 3D accelerometer 706 may also be used to capture micro-motions of the hand of a user holding the electronic device 700. In this case, the 3D accelerometer data is captured by the accelerometer 706, sampled, pre-processed, and then provided to the NFP certification controller.

[0126] If the accelerometer is not available in the electronic device, one or both of the button / pad 704 and the keypad 705 are touch sensitive in three dimensions including X, Y, and Z for NFP authentication. The Z axis is the axis perpendicular to the button and keypad and pressure can be applied by a finger to select the potential function of the button or key of the keypad to control the electronic device. The button / pad 704 can be, for example, a power button used to turn on / off the electronic device. The button / pad 704 can be, for example, a home button that causes the electronic device to enter a "home" or initial user interface state.

[0127] One or both of the touch-sensitive buttons / pad 704 and / or the touch-sensitive keypad 705 may be used to capture three-dimensional raw micro-motion signals that may be used for NFP authentication.

[0128] If the display device 707 is a touch-sensitive display device, the keypad 705 or the button 704 may be displayed on the touch-sensitive display device 707 from which three-dimensional raw micro-motion signals may be captured.

[0129] NFP certification controller part (see Figures 8A to 8C , the NFP authentication controllers 810A to 810C, 810 in 14A may be formed by the processor 701 executing instructions re-called from the firmware / software 712 stored in the storage device 702.

[0130] Reference Figure 7 and 8A , shows a functional block diagram of a portion of an NFP authentication system having a three-dimensional touch-sensitive button / pad 704. The NFP authentication system further includes an NFP authentication controller 810A coupled to the touch sensor 801 of the three-dimensional touch-sensitive button / pad 704.

[0131] The three-dimensional touch-sensitive button / pad 704 includes a 3D touch sensor 801 proximate the surface of the pad 704 for sensing position and changes in X, Y finger position and finger pressure Z. The three-dimensional touch-sensitive button / pad 704 typically includes a function button switch 814 for generating function control signals sent to the processor. The 3D touch sensor 801 generates a raw three-dimensional displacement signal including micro-motion sensed at the user's finger 890 and indicated by the 3D arrow 850.

[0132] After some pre-processing, the NFP authentication controller 810A receives data samples representing micro-motions sensed by the touch sensor 801 and generates the user's NFP. The calibration NFP file of the authorized user calibration parameters is usually stored in a non-volatile memory or other non-volatile storage device within the NFP authentication controller or in a non-volatile memory or other non-volatile storage device coupled to the NFP authentication controller or coupled to communicate with the NFP authentication controller (e.g., a storage device of the memory 702 or a server). In response to the NFP and the stored user calibration parameters trained by the authorized user and associated with the authorized user, the NFP authentication controller 810A classifies the NFP and generates a match percentage value. In response to the match percentage value and the predetermined acceptable match percentage, the NFP authentication controller 810A may generate an access grant signal 1449 granting access to the electronic device to the authorized user who is touching the touch sensor. The NFP authentication controller 810A may be a functional process that a processor (e.g., processor 701) is configured (by hardware, software, or a combination of hardware and software) to perform.

[0133] Reference Figure 7 and 8B , shows a functional block diagram of an NFP authentication system with a three-dimensional touch-sensitive keypad 705. The touch-sensitive keypad 705 includes a matrix of an M×N array of 3D touch sensors 801AA to 801MN, each of which can sense the position and changes in X, Y finger position and finger pressure Z of a finger 890 and generate raw micro-motion data. Each pad can include a function button switch 814 used to generate multiple function control signals while capturing raw micro-motion sensor data by the 3D touch sensors 801AA to 801MN.

[0134] The signals from the M×N array of 3D touch sensors 801AA to 801MN are pre-processed and sampled, where the sampled data is coupled to the NFP authentication controller 810B. In granting access, the NFP authentication controller 810B behaves similarly to the NFP authentication controller 810A. However, the NFP authentication controller 810B can be trained to support slightly different signals that can be expected from different touch sensors in the matrix of the M×N array of 3D touch sensors 801AA to 801MN.

[0135] Therefore, for each different touch sensor, the calibration NFP file of the authorized user calibration parameters may be different. It is expected that for the user calibration parameters of each of the multiple touch sensors, the calibration NFP file of the authorized user calibration parameters may be larger. In the case of a larger calibration NFP file, the non-volatile memory or other non-volatile storage device storing the file may be slightly larger and have a slightly larger capacity. Similarly, a non-volatile memory or other non-volatile storage device (e.g., a storage device of the memory 702 or a server) coupled to the NFP certification controller or coupled to communicate with the NFP certification controller may have a larger storage area and / or capacity for accommodating slightly larger NFP calibration files from one or more users. The NFP certification controller 810B may be a functional process that a processor (e.g., processor 701) is configured (by hardware, software, or a combination of hardware and software) to execute.

[0136] Reference Figure 7 and 8C , a functional block diagram of an NFP authentication system having a three-dimensional accelerometer 706 in an electronic device 700 is shown. A user hand 899 holds the electronic device 700 including the three-dimensional accelerometer 706. The three-dimensional accelerometer 706 is coupled to an NFP authentication controller 810C. The NFP authentication controller 810C may be a functional process that a processor (e.g., processor 701) is configured (by hardware, software, or a combination of hardware and software) to perform.

[0137] When the electronic device 700 is held stably in his / her hand 899, micro-motions in the user's hand 899 and indicated by the 3D arrow 852 are sensed by the 3D accelerometer 706. As the electronic device 700 held in the user's hand is moved to adjust the position (e.g., from a pocket to the user's ear), undesirable micro-motions in addition to the micro-motions are also sensed by the 3D accelerometer 706. As further explained herein, these undesirable micro-motions will be suppressed, filtered out, or eliminated in the desired signal.

[0138] Touch sensors are typically different from 3D accelerometers. Therefore, the calibration NFP files of authorized user calibration parameters will likely be different. Some calibration data of filter parameters of a digital filter generated to filter out unwanted motion (e.g., micro-motion) in a three-dimensional accelerometer signal from the 3D accelerometer may be included as part of an NFP calibration file associated with a user. Therefore, the NFP calibration file of the authorized user calibration parameters of a 3D accelerometer may be different from the NFP calibration file of the authorized user calibration parameters of a touch sensor. In either case, the NFP calibration file of the authorized user calibration parameters is typically stored in a non-volatile memory or other non-volatile storage device within the NFP certification controller or in a non-volatile memory or other non-volatile storage device coupled to the NFP certification controller or coupled and the NFP certification controller communicates (e.g., a storage device of memory 702 or a server).

[0139] The raw three-dimensional accelerometer signal generated by the 3D accelerometer 706 is pre-processed and undergoes compensation due to gravity. The raw three-dimensional accelerometer signal is sampled into data samples of a data set so that digital filtering can be performed using digital signal processing and using a digital processor (e.g. Figure 7 The processor 701 shown in FIG. 810 performs the digital conversion. The data samples of the accelerometer data are coupled to the NFP certification controller 810C.

[0140] The NFP authentication controller 810C receives data samples representing micro-motions sensed by the accelerometer 706 and generates an NFP for the user. In response to the NFP and the stored user calibration parameters trained by and associated with the authorized user, the NFP authentication controller 810C classifies the NFP and generates a match percentage value. In response to the match percentage value and a predetermined acceptable match percentage, the NFP authentication controller 810C may generate an access granting signal 1449 granting access to the electronic device to the authorized user holding the electronic device.

[0141] Although a three-dimensional sensor (e.g., a 3D touch sensor, a 3D accelerometer, or a 3D motion sensor) used to capture three-dimensional micro-motion signals as three-dimensional signals used to generate NFP has been described herein, the sensor may be multi-dimensional to capture a multi-dimensional signal having at least two dimensions. For example, a touch-sensitive surface may capture a variable impedance having two variable dimensions of resistance and capacitance that may be used to generate a three-dimensional representation of NFP from a variable two-dimensional signal.

[0142] Signal Processing

[0143] For temporary reference Fig.14AIn order to obtain NFP 1460 from the sampled micro-motion signal 1450 captured by the sensor and sampled by the sampler (A to D converter), several signal processing steps are performed by the processor in the electronic device. The signal processing steps performed on each dimension in the sampled micro-motion signal 1450 are performed by the signal processing and feature extraction module 1401 of the NFP certification system 810. These one or more signal processing algorithms performed by the module 1401 may generally be referred to herein as NFP algorithms and methods.

[0144] Typically, a sequence of events is performed to obtain the user's NFP and then evaluate its authenticity.

[0145] A raw data file (in the X, Y and Z directions) is obtained from the 3D accelerometer or one or more touch pad sensors (as the case may be) over a predetermined sampling period.

[0146] The raw data file is sampled over a predetermined time span (e.g., 5, 10, 20, or 30 seconds) at a predetermined sampling frequency to capture the signal of interest that is compatible with the desired further filtering (e.g., 250 Hz (4 milliseconds between samples), 330 Hz, 200 Hz, or as low as 60 Hz (twice the 30 Hz frequency of interest)).

[0147] Signal processing is performed on the sampled signal to generate a micro-motion signal having a specific frequency component of interest. The sampled signal is filtered using a bandpass filter having a frequency range between 7 to 8 Hz, 7 to 12 Hz, or 8 to 30 Hz. Micro-motion signals in this frequency range are most useful for distinguishing users. The bandpass filter may also suppress large amplitude signals due to autonomous or involuntary movement that may be captured by a sensor (e.g., an accelerometer). If an accelerometer is used as the sensor, the effect of gravity is compensated or removed by signal processing. If an accelerometer is used as the sensor, the sampled signal is made position-invariant or orientation-invariant by signal processing. The micro-motion signal is a position-invariant sampled signal that can be consistently used to extract feature values ​​for comparison between users.

[0148] Additional signal processing is performed on the micro-motion signal to generate a signal processed waveform signal from which the value of the NFP can be extracted. A predetermined feature is selected whose value will be extracted to represent the NFP. The value of the NFP can be extracted directly from the signal processed waveform signal, each micro-motion signal, and / or from both after performing additional signal processing. In any case, a unique value representing a unique NFP for the user is extracted, which unique NFP will be different from other NFPs generated by other users.

[0149] The user's NFP can be used in several different applications. A classifier (various data mining techniques can be used as the classifier) ​​is used to generate a match result value (e.g., a percentage). The classifier is trained / calibrated using an initial NFP (calibration NFP) that generates authorized user calibration parameters so that a calibrated match result level is achieved. Thereafter, the classifier can be used in user mode in conjunction with the authorized user calibration parameters. The classifier in the user mode generates a match result value to authenticate the user as an authorized user or an unauthorized user. In response to a predetermined access match level, the authentication controller can determine whether the user is an authorized user based on the match result value.

[0150] Several of these signal processing steps are described in further detail herein.

[0151] Signal sampling

[0152] Reference Fig. 9 , showing a hand acceleration waveform 900 of a hand acceleration signal of a single axis (X, Y or Z) varying with time. Portion 901 of the hand acceleration waveform 900 is magnified as waveform 900T, as shown. Although an analog signal waveform may be shown in the figure, it should be understood that the analog signal waveform may be sampled over time and represented by a sequence of numbers with discrete periodic time stamps ("digital waveform"). Although the accelerometer senses acceleration that varies with time, if the sensor instead senses displacement that varies with time, the displacement may be converted into acceleration by secondarily differentiating the displacement signal that varies with time.

[0153] The hand acceleration of each axis is sampled within a predetermined sampling time period 905, such as, for example, a 10, 20 or 30 second time span. The sampling frequency is selected so that it is compatible with subsequent filtering. For example, the sampling frequency may be 250 Hz (4 milliseconds between samples). Alternatively, the sampling frequency may be, for example, 330 Hz or 200 Hz. The analog signal is sampled by a sampling analog / digital converter to generate samples S1 to SN represented by a digital number within timestamps T1 to TN during a given predetermined sampling time period. Assuming a 20 second sampling time period and a 250 Hz sampling frequency, the acceleration data set will contain 3 (3 axes) times 5000 samples within a time period for a total of 15k samples.

[0154] Signal Normalization for Uncorrelated Signals

[0155] The generation of NFP is based on three-dimensional uncorrelated signals. The 3D accelerometer used to sense tremor is part of an electronic device held by the user's hand. Fig. 10A, the device 700 and the device axes Xd, Yd, Zd may be held by the user's hand in different orientations relative to the world W and the world axes Xw, Yw, Zw. Therefore, the raw sensor data from the 3D accelerometers of the device axes Xd, Yd, Zd are correlated. Therefore, the raw sensor data from the 3D accelerometers of the device axes Xd, Yd, Zd depends on the orientation of the electronic device 700. A signal normalization process is performed using principal component analysis (PCA) to make the 3D accelerometer data for each axis orientation-invariant or rotation-invariant and therefore uncorrelated.

[0156] Each data set of raw sensor data from a 3D accelerometer of device axes Xd, Yd, and Zd within a given sampling time period is analyzed in a 3D feature space. Three eigenvectors and eigenvalues ​​are determined for each axis. The eigenvalues ​​of each eigenvector are compared to determine the maximum eigenvalue that identifies the maximum eigenvector. Next, the points of the data set are rotated (the points and their data values ​​are transformed in space) so that the maximum eigenvector is aligned with a predetermined axis and along a predetermined axis. For all data sets of 3D raw sensor data originating from the same electronic device 700, the predetermined axis is constant. For any device that implements the NSP algorithm using a 3D accelerometer, the predetermined axis may even be constant. The predetermined axis may be, for example, the Zw world axis. Aligned along the predetermined axis, the maximum eigenvector of each data set transforms the X, Y, and Z components of the 3D point in the raw sensor data so that they are uncorrelated and rotationally invariant.

[0157] For example, Fig. 10B 3D point 1011A illustrating acceleration relative to eigenvector 1010A. The data set forming eigenvector 1010A (including point 1011A) is rotated in 3D space to eigenvector 1010B aligned with the Zw world axis. Therefore, point 1011A and its X, Y, Z component values ​​are transformed in 3D space to point 1011B and its X', Y', Z' component values.

[0158] Signal suppression / extraction / filtering

[0159] This NFP algorithm specifically extracts tremors or micromovements from transformed sensor data signals related to motor control of the cerebral cortex, its subcortical parts, cerebellum, and central nervous system that are affected or not affected by peripheral structures (e.g., muscles, bones, glands, etc.), and suppresses unwanted parts of the transformed sensor data signals.

[0160] Therefore, it is desirable to generate or extract micro-motion signals from the transformed hand acceleration signal. However, the transformed hand acceleration signal may have several undesirable signals therein that may be suppressed, removed, or filtered out. For example, the hand holding an electronic device often moves with large movements, such as moving the electronic device from a person's pocket / purse or pocket so that a keypad is accessible and a display screen is visible.

[0161] Large acceleration swings 902 may occur due to such large movements, such as Fig. 9 In some cases, these large swings of the signal from the user's large movements (micro-movements) can be used as a behavioral profile, such as described in U.S. Patent Application No. 13 / 823,107, filed by Geoff Klein on January 5, 2012, entitled "METHOD AND SYSTEM FOR UNOBTRUSIVE MOBILE DEVICE USER RECOGNITION". However, since these large swings are not closely related to the micro-movements associated with neuromuscular tremors, it is desirable in this case to suppress or remove these large swings during signal processing and generation of a unique NFP for the user.

[0162] Unwanted signal components may be suppressed, for example, due to large swings 902. The large swings may come from vibrations associated with the building or structure in which the user is located or from the movements that the user actually performs when holding the electronic device with the acceleration sensor.

[0163] Most buildings and structures resonate at frequencies between 3 Hz and 6 Hz. Therefore, vibrations from buildings and structures are outside the desired frequency range (e.g., outside the range of 8 Hz to 30 Hz) and are unlikely to resonate and contaminate the desired signal with that range. Signals in the range of 3 Hz to 6 Hz will then be filtered out.

[0164] Large-scale movements (large movements) of the user include jumping with the electronic device, moving one's arm with the electronic device, turning with the electronic device, walking, running, jogging and other large body movements with the electronic device. Large-scale movements (large movements) of the user are usually not conducive to the generation of NFP. The NFP algorithm is not based on hand or body gestures, nor is NFP based on a motion spectrum / library.

[0165] Large-scale movements of users (large movements) form, for example, Fig. 9The large-scale signal swing 902 shown in FIG. The large-scale motion of the user is unlikely to be repeated when attempting to regenerate the NFP during the authentication procedure. The large-scale motion of the user may adversely skew the algorithm or calculation in generating the NFP. Therefore, the large-scale signal from the large-scale motion is partially suppressed or filtered out from the acquired data by bandpass filtering performed for the desired frequency range. Over a relatively long period of time, the large-scale signal from the large-scale motion is highly correlated. The three-dimensional micro-motions attributed to the nervous system are not closely correlated. Therefore, subsequent signal processing used to make the three-dimensional signal uncorrelated will suppress the large-scale signal from the large-scale motion.

[0166] Alternative means of suppressing / filtering out large-scale signals from the desired signal, due to large-scale motion of the user, may also be used. The electronic signal may be analyzed and then classified / identified as small and large signals separated from the small signal amplitudes of micro-motion. The analysis may be of the form described in Richard J. Povinelli et al., "Time Series Classification Using Gaussian Mixture Models of Reconstructed Phase Spaces," IEEE Transactions on Knowledge and Data Engineering, Vol. 16, June 6, 2004. Alternatively, large signals due to voluntary movements can be separated by using a BMFLC-Kalman filter as described in the document attached in the Appendix: "Estimation of Physiological Tremor from Accelerometers for Real-Time Applications" by Kalyana C. Veluvolu et al., Sensors, 2011, Vol. 11, pp. 3020-3036.

[0167] The amplified hand acceleration waveform 900T is more representative of an acceleration signal including tremors from which micro-motion (micro-acceleration) signals of interest may be generated. The waveform 900T has several frequency components outside of a frequency range of interest. For example, the frequency range of interest in the signal is from 8 Hz to 12 Hz and / or from 8 Hz to 30 Hz. These frequency ranges are associated with known tremors that most people should typically have.

[0168] Reference Fig.11A, shows a bandpass filter (BPF) with a filter response with a lower cutoff (LC) frequency and an upper cutoff (UC) frequency at the ends of a desired frequency range. The BPF is in the form of a digital finite impulse response (FIR) bandpass filter used to filter digital signals. This BPF will filter out undesirable frequency signal components from the raw electronic signal captured by the sensor to produce a micro-motion acceleration signal of interest in the desired frequency range. Three BPFs can be used in parallel, one for each axis signal. Alternatively, one BPF can be time-shared between each axis.

[0169] Reference Fig. 11B , a digital high-pass filter (HPF) with an LC frequency and a digital low-pass filter (LPF) with an UC frequency can be combined in series to achieve a result similar to a digital FIR bandpass filter in generating a micro-motion acceleration signal of interest. Alternatively, a low-pass filter with an UC frequency and a high-pass filter with an LC frequency can be combined in series to achieve a similar synthetic signal output.

[0170] Additional signal processing is performed on the micro-motion signal using a processor to generate the NFP.

[0171] Gravity correction for accelerometer sensors

[0172] It is desirable to remove the effects of gravity on the three-dimensional accelerometer data captured by the accelerometer. As such, the generated NFP is substantially independent of the device orientation. A dedicated three-dimensional touch pad sensor may not be affected by gravity. In this case, the three-dimensional data captured by the touch pad sensor when a user's finger touches it does not need to follow these steps.

[0173] Gravitational acceleration is a constant. Therefore, the effect of gravity on the spectrum is expected to be a fixed vector close to zero frequency. Gravity acts as a zero frequency DC component, so one would expect it to achieve leakage towards low signal frequencies (below 1 Hz).

[0174] Reference Fig. 12A A high pass filter with a cutoff frequency below 1 Hz (e.g., 0.25 Hz) can be used to compensate for gravity and improve the accuracy of recognition using NFP. The micro-motion signal with the effect of gravity is coupled into a gravity high pass filter (HPF) with a high pass filter response, where the cutoff (CO) frequency is less than 1 Hz but greater than 0 Hz. The signal output from the gravity HPF is a micro-motion signal without the effect of gravity.

[0175] Reference Fig. 12BAnother way to eliminate the effect of gravity is to transform the coordinates of the 3D acceleration data points (ADP1 to ADPN) of each data set. First, the XYZ coordinates of the center of gravity CG of the sampled data set of 3D acceleration points (X, Y, Z signals) within a predetermined sampling period are determined. Next, the coordinates of the 3D acceleration points in a given data set undergo a translation transformation so that the center of gravity is at the (X=0, Y=0, Z=0) coordinate or axis origin. In this case, the entire data set undergoes a translation transformation.

[0176] Yet another way to eliminate the effects of gravity on the acceleration signal would be to analytically correct the gravity vector for the phase shift induced in the 3D accelerometer data. The gravity vector may be determined from the raw 3D acceleration signal. The 3D accelerometer data may then be attenuated in one direction responsive to the gravity vector and enhanced in the opposite direction of the gravity vector.

[0177] Filtered motion waveform

[0178] After filtering and suppression to remove unwanted signals and transforming to compensate for gravity or relation, the desired micro-motion signal is formed.

[0179] Reference Fig.13A , the waveform diagram shows three axes of raw acceleration waveform data 1301 to 1303 offset from one another due to gravity and orientation differences. Fig. 13B Illustrate acceleration waveform data for three axes after suppressing / removing / filtering unwanted signals from the captured acceleration sensor data. Fig. 13B The acceleration waveform data in represents the micro-motion signal waveform. If a 3D touch sensor is used instead of a 3D accelerometer, the displacement waveforms in the three axes can be a composite signal. Figures 13A to 13B Four seconds of sampling are shown, but sampling periods of greater or lesser time may be used.

[0180] There are three axes of micro-motion signals corresponding to the three axes of acceleration sensed by the 3D accelerometer or the three axes of displacement sensed by the 3D touch pad sensor. The three axes of the micro-motion signals define plottable three-dimensional points x(t), y(t), z(t). The three-dimensional micro-motion signals x(t), y(t), z(t) of the user may be further processed into phase(t), y(t), z(t) within a sampling time period and plotted, for example, on a Figures 2A to 2C The three-dimensional Poincare phase scatter plot shown in .

[0181] The three-dimensional Poincare phase scatter plots of different users show that the micro-motion signal has a unique pattern that can be used to identify and authenticate the user. The unique pattern for each user can be obtained from the Poincare phase scatter plot. Another unique pattern for each user can be obtained from the time series of the micro-motion itself without any phase information. However, it is easier to use a signal processor and digital signal processing algorithms to extract the unique pattern from the signal itself.

[0182] NFP certified controller

[0183] Reference Fig.14A , showing a block diagram of an NFP authentication controller 810. The NFP authentication controller 810 includes a signal processing and feature extractor module 1401 (which may be divided into two separate modules), an NFP authentication classifier module 1402, and an authorization (authentication) controller module 1404. The NFP authentication controller 810 may further include an optional secondary authentication module 1406 for multi-factor authentication, such as by a keypad. One or more of the modules may be implemented by software / firmware instructions executed by a processor, hard-wired electronic circuits, or a combination of each.

[0184] The NFP authentication controller 810 may further include: a non-volatile storage device 1453 coupled to the classifier 1402 and an authorization (authentication) controller module 1404 for storing data (e.g., user calibration parameters 1466); an access match (AM) level 1456A, a mandatory recalibration (MR) level 1456B, an autonomous recalibration (VR) level 1456C (collectively referred to as the match percentage level 1456); and an authentication enable bit (EN) 1455. Alternatively, the non-volatile storage device 1453 may be external to the NFP authentication controller 810, but maintained within the electronic device as a secure independent non-volatile storage device or a secure portion of a larger non-volatile storage device.

[0185] The NFP certification controller 810 receives three dimensions of micro-motion data samples 1450 in each data set within each sampling period. The micro-motion data samples 1450 are coupled into the signal processing and feature extractor module 1401. The signal processing and feature extractor module 1401 performs signal processing and signal analysis on the micro-motion data samples 1450 to extract a plurality of extracted features 1460X, 1460Y, 1460Z for each of the respective three dimensions (X, Y, Z). The extracted features 1460X, 1460Y, 1460Z collectively represent a NeuroFingerPrint (NFP) 1460 coupled into the NFP certification classifier module 1402.

[0186] The NFP authentication classifier module 1402 receives the NFP 1460 (extracted features from the micro-motion data sample 1450) and generates a match percentage (MP) signal output 1465. In the user mode, the match percentage signal 1465 is coupled to the authentication controller 1404. In the calibration or training mode, the match percentage signal 1465 is used by a processor in the electronic device to evaluate the match percentage signal 1465 in response to the selection of the initial user calibration parameters 1466. The duration of the training / calibration of the NFP authentication classifier module and the generation of the initial user calibration parameters 1466 is expected to be between 5 and 10 seconds. It is expected that in the user mode, it takes less than 5 seconds to sense the movement in the user's body part and determine whether to grant or deny the access.

[0187] In the calibration or training mode, an authorized user may generate one or more sets of user calibration parameters 1466 so that the NFP authentication system operates under different conditions. For example, the user may want to hold the electronic device in either hand and have been granted access. The tremor will be different between the left and right hands or between different fingers. The user may want to calibrate the NFP authentication system to both or multiple fingers of the left and right hands. In addition, in some cases, more than one user will use the electronic device. In this case, multiple people may be authorized users and multiple calibrations will need to be performed and stored. Therefore, the storage device 1453 can store multiple sets of user calibration parameters for the same authorized user or different authorized users.

[0188] An authentication enable bit (EN) 1455 is coupled to the authentication controller 1404. The authentication enable bit (EN) 1455 may be used to enable the authentication controller 1404 after an initial calibration or training mode. After the initial calibration or training mode, the authentication enable bit (EN) 1455 is set and the authentication controller 1404 is enabled. After the initial calibration or training mode, the authentication enable bit (EN) 1455 is not reset in the user mode or recalibration mode. Unless the entire electronic device is erased along with the enable bit, the authentication controller 1404 is enabled by the enable bit 1455 for enhanced security.

[0189] In the user mode, the match percentage signal 1465 is used by the authentication controller 1404 to evaluate whether to grant access to the electronic device and its software applications. The match percentage signal 1465 is evaluated with the access match level 1456A to generate an access grant (AG) signal 1499. If the match percentage signal 1465 is greater than or equal to the access match level 1456A, then the access grant (AG) signal 1499 at a logic level is generated to signal that access has been granted. If the match percentage signal 1465 is less than the access match level 1456A, then the access grant (AG) signal 1499 is not generated and access is not granted. The access grant (AG) signal 1499 is coupled to the processor to enable an authorized user to control and operate the functions of the electronic device.

[0190] If the optional secondary authentication mode is used, a secondary match (SM) signal 1468 is coupled into the authentication controller 1404. In this case, the authentication controller 1404 further evaluates the secondary match signal 1468 as to whether to generate an access grant (AG) signal 1499. The authentication controller 1404 may use AND logic to require that two conditions be met before generating the access grant signal. Alternatively, the authentication controller 1404 may use OR logic to require that two conditions be met before generating the access grant signal.

[0191] In response to the inactive reactivation signal 1470 from the processor, the authentication controller 1404 maintains the access-granted signal 1499 in an active state (as long as the user uses the electronic device) and avoids a sleep state or timeout to enter a protected state. If the sleep state or timeout occurs, the reactivation signal 1470 is pulsed by the processor. In response to the pulsed reactivation signal, the authorization (authentication) controller module 1404 deactivates the access-granted signal 1499 so that the user must reauthenticate himself / herself using the NFP authentication system of the electronic device to obtain access.

[0192] Reference Fig.15 , shows a graph of the match percentage level (MP) 1456 compared to the match percentage signal 1465 along the X-axis. The Y-axis indicates the generation of the access grant (AG) signal 1499 by the authentication controller 1404 and whether access is granted to the suspicious user.

[0193] In the user mode, the match percentage signal 1465 is used by the authentication controller 1404 to evaluate whether to grant access to the electronic device and its software applications. The match percentage signal 1465 is evaluated against the access match level 1456A. ​​If the match percentage signal 1465 is at or above the access match level 1456A, an access grant (AG) signal 1499 is generated by the authentication controller 1404. If the match percentage signal 1465 is below the access match level 1456A, the authentication controller 1404 does not generate the access grant (AG) signal 1499.

[0194] Reference Fig.14A and 15 , the authentication controller 1404 may also generate one or more recalibration signals to notify the user to recalibrate the NFP authentication system by regenerating the user calibration parameters 1466 for the authorized user before the NFP authentication classifier 1402 fails to generate a match percentage signal 1465 at or above the access match level 1456A.

[0195] In the training or calibration mode, the NFP certification classifier 1402 is trained / calibrated by the initial user calibration parameters 1466 so that the NFP certification classifier 1402 produces a level of matching percentage signal 1465 at a calibration level 1457 that may be or approach 100% (e.g., 98%). After training, in the user mode, the NFP produced by the user over time may vary with his / her body age, disease, or other reasons affecting the physiological condition of the body. Therefore, the matching percentage signal 1465 may decrease from the calibration level 1457 over time. Resetting the user calibration parameters 1466 using periodic recalibration restores the matching percentage signal 1465 to the calibration level 1457. Depending on the user's age, health, and other physiological conditions of the body, the user may generally require more or less periodic recalibration.

[0196] In the user mode, the NFP authentication classifier 1402 is used by the authentication controller 1404 to evaluate whether recalibration is required. The determination of recalibration is in response to the mandatory recalibration (MR) level 1456B and the autonomous recalibration (VR) level 1456C. The mandatory recalibration (MR) level 1456B is less than both the autonomous recalibration (VR) level 1456C and the calibration level 1457. The autonomous recalibration (VR) level 1456C is less than the calibration level 1457.

[0197] Recalibration requires that the user be authenticated as an authorized user first, and access to the electronic device must first be authorized to enter the user mode and recalibrate. If the device is stolen by an unauthorized user, the unauthorized user will not be granted access to recalibrate the device.

[0198] If the match percentage signal 1465 drops below the autonomous recalibration (VR) level 1456C, an autonomous recalibration signal 1471 is generated by the authentication controller 1404. In this case, the electronic device notifies the user through its user interface that he / she should pause and take some time to autonomously recalibrate the NFP authentication system by regenerating the user calibration parameters 1466. It is expected that most users will autonomously choose to recalibrate the NFP authentication system. However, some authorized users will choose to wait, forget the warning, or ignore the warning altogether.

[0199] For those authorized users who do not voluntarily recalibrate the NFP authentication system, they may be forced to undergo a mandatory recalibration process. If the match percentage signal 1465 further decreases and falls below the mandatory recalibration (MR) level 1456B, a mandatory recalibration signal 1472 is generated by the authentication controller 1404. In this case, after the authorized user is authenticated by the NFP authentication classifier 1402 and access is granted by the authentication controller 1404, the electronic device immediately enters a recalibration mode, notifying the user through its user interface to prepare and execute the recalibration procedure by continuing to hold the device appropriately or touching a button appropriately. The user is further notified that the NFP authentication system is performing a recalibration and continues to wait until the recalibration is complete and that the NFP authentication system has been successfully recalibrated as the user calibration parameters 1466 are regenerated to generate a match percentage signal 1465 at or above the calibration level 1457.

[0200] If the electronic device is not used for a period of time, the match percentage signal 1465 is likely to further decrease and fall below the access match (AM) level 1456A. ​​In this case, an authorized user may be denied access. Therefore, the selection of the value of the access match (AM) level 1456A is important so that authorized users are not easily denied access to the electronic device, while unauthorized users are denied access. It is desirable to generate an NFP signal 1460, which is generated by module 1401 and coupled to the NFP authentication classifier 1402, that is less sensitive to changes in the user (e.g., aging, disease) so that recalibration is less frequent. Therefore, it is desirable to select the signal processing and feature extraction algorithm of module 1401 so that the sensitivity in generating the NFP is low and is unlikely to change over time.

[0201] An example of the match percentage levels (MP) 1456 in increasing order are set as follows: the access match (AM) level 1456A is 85%, the mandatory recalibration (MR) level 1456B is 90%, and the autonomous recalibration (VR) level 1456C is 95%. Thus, the user is notified by the user interface of the electronic device to perform an autonomous recalibration before a mandatory recalibration needs to be performed.

[0202] In the event that a mandatory recalibration is required by the user, the NFP authentication system should be prevented from denying access to the electronic device if the electronic device is actively used. If the electronic device is left unused for a period of time (e.g., one or more years), the user may be required to wipe the electronic device clean, reinitialize the NFP authentication system, and reload applications and / or data, such as from a backup.

[0203] Reference Figures 14A to 14B , showing a model for the NFP certification classifier 1402. According to one embodiment, the model is a regression analysis model that may be linear. In an alternative embodiment, the regression analysis model for the NFP certification classifier 1402 may be non-linear. During the training or calibration mode, features of micro-motion signals from authorized users are extracted by the signal processing and feature extraction module 1401 to generate a calibration NFP 1460, including calibration NFP information 1460X, 1460Y, 1460Z for each axis.

[0204] The calibration NFP information 1460X, 1460Y, 1460Z for each axis is placed in a single row NFP matrix 1452. The appropriate value of the authorized user calibration parameter 1466 is unknown. The value of the user calibration parameter 1466 can be randomly set into the single column calibration matrix 1454. Matrix multiplication is performed by the processor to multiply the single row NFP matrix 1452 and the single column calibration matrix 1454 to obtain the match percentage value 1465.

[0205] During calibration / training, the processor searches out user calibration parameters 1466 that will be placed in the single column calibration matrix 1454 to produce a predetermined value for the match percentage 1465. This predetermined value is called the calibration level 1457. For example, the calibration level 1457 can be set to 90%. In this case, the processor searches out the value of the authorized user calibration parameters 1466 so that when multiplied by the NFP 1460, the match percentage 1465 output from the classifier is 90% or greater.

[0206] Once the authorized user calibration parameters 1466 are set and stored in the electronic device, the NFP classifier 1402 uses the authorized user calibration parameters 1466 to multiply subsequent regenerated NFPs generated from unknown users or authorized users.

[0207] If a person is subsequently unable to produce a regenerated NFP substantially similar to the calibrated NFP, the value of the match percentage 1465 produced by the classifier 1402 will be low when the user calibration parameter 1466 is multiplied by the different regenerated NFP. The authorization (authentication) controller module 1404 can be set so that a low value of the match percentage 1465 denies access to unknown persons who produce significantly different regenerated NFPs.

[0208] If a person subsequently generates an NFP (a regenerated NFP substantially similar to the calibration NFP), the value of the match percentage 1465 generated by the classifier 1402 will be high when the user calibration parameter 1466 is multiplied by the regenerated NFP. The authorization (authentication) controller module 1404 may be configured so that a high value of the match percentage 1465 above the access match level 1456A grants access to the electronic device.

[0209] Signal processing and feature extraction for generating NFP

[0210] Reference Fig.13A , the figure shows raw, unfiltered samples of three-dimensional acceleration data from a 3D accelerometer. For the X, Y, and Z axes of the electronic device, raw signals 1301-1303 are captured for each of the three dimensions. With some pre-processing, bandpass filtering, and gravity compensation, Fig. 13B The micro-motion signal waveforms 1311 to 1313 shown in FIG. 1 may be formed from the original signals 1301 to 1303, respectively. Further signal processing may then be performed on the micro-motion signal waveforms 1311 to 1313 to form an NFP for a given sample of acceleration data. There are various ways to generate an NFP from a micro-motion signal. Once calibrated / trained, the NFP authentication system in the electronic device uses a consistent method for generating the NFP.

[0211] The three-dimensional micro-motion signal is related to the common tremor present in the user. There are patterns hidden in the time domain of the micro-motion signal, which are unique to each user because they originate from his / her neuromuscular function. It is desirable to highlight the pattern by performing signal processing on the micro-motion signal and then extracting features from the micro-motion signal to form an NFP. The calibration NFP can then be used during the training or calibration process to generate authorized user calibration parameters. For the purpose of user authentication, the authorized user calibration parameters can then be used to classify the reproduced NFP and distinguish known authorized users from unknown unauthorized users.

[0212] However, before the NFP can be used for authentication, the NFP is used to train or calibrate the model of the classifier by generating a set of authorized user calibration parameters. This set of authorized user calibration parameters is then used in conjunction with the classifier model to evaluate future NFPs (regenerated NFPs) captured by the sensor from unknown persons. Various signal processing algorithms can be used to extract data from the micro-motion signals as NFPs.

[0213] Inverse spectrum analysis can be used to detect patterns hidden in micro-motion signals, e.g. Fig. 13B The micro-motion signal waveforms 1311 to 1313 shown in FIG. The inverse spectrum analysis is performed to recover the unknown signal components related to the time variation.

[0214] One type of inverse spectrum analysis that can be used is cepstrum analysis. Cepstrum analysis is a tool for detecting the periodicity of a spectrum. The cepstrum analysis can be used to detect repeating patterns, their periodicity, and frequency spacing.

[0215] Typically, the cepstrum is the result of taking an inverse Fourier transform (IFT) of the logarithm of the estimated spectrum of the signal.

[0216] As shown by the following equation, the power cepstrum of a signal may be defined as the square magnitude of the inverse Fourier transform of the logarithm of the square magnitude of the Fourier transform of the signal.

[0217]

[0218] First, a fast Fourier transform is used to obtain the Fourier spectrum of a micro-motion signal (such as the micro-motion signal shown in FIG. 13 ). Mathematically, this can be accomplished by performing a Fourier transform using the following equation:

[0219]

[0220] Where ξ represents the real frequency (in Hertz) value and the independent variable x represents time, which is the transformed variable. Using software or digital circuits that can digitally sample the signal, a discrete time Fourier transform (DFT) can be used, represented by the following equation:

[0221]

[0222] The result of the Fourier transform of the micro-motion signal is the spectral density (power spectrum or Fourier spectrum). Fig.16 An example of a spectral density presented for a tremor at the hand, over time, is illustrated. It should be noted that the expected peak (P) signal component for a physiological tremor is a peak frequency (PF) at about 10 to 12 Hz. However, there is still a lot of useful hidden pattern (HP) information in the spectral density curve for tremor.

[0223] The spectral density (power spectrum) is the composite of the component frequencies of a signal. That is, there are many signal frequencies that combine together to form the spectral density curve. It is desirable to effectively separate the signal frequencies to reveal a pattern. The signal composite described by the spectral density is a signal convolution equivalent to the product of the constituent signals.

[0224] Taking the logarithm of the convolution effectively turns the spectral density into a sum rather than a product of constituent signals. Furthermore, the logarithm of the spectral density (or the logarithm of the square of the spectral density) highlights lower amplitude frequency components in the spectral density curve where hidden pattern (HP) information may exist. Taking the logarithm of the spectral density effectively compresses large signal amplitudes of the spectral density signal and expands smaller amplitudes of the spectral density signal.

[0225] However, it is difficult to see the periodicity and pattern of the signal waveform produced by the logarithmic transformation. Therefore, an inverse Fourier transform (IFT) is performed on the logarithmic signal waveform so that the unknown frequency components and hidden patterns are visible. The inverse Fourier transform of the logarithmic transform separates the components of the complex signal in the spectral density of the micro-motion signal waveform.

[0226] Figures 17A to 17B illustrate Fig. 13B The cepstrum presented by the micro-motion waveform shown in . This synthetic waveform is generated after performing an inverse Fourier transform (IFT) of the spectral logarithm. The cepstrum produces a series of roughly resolved unknown signal components in the synthetic waveform. The value of the characteristic pattern can be extracted from each axis (dimension) in the synthetic waveform and used as the NFP for identifying the user. Note that Figures 17A to 17B The horizontal axis of the cepstrum waveform shown in is frequency and not a time measure in the time domain.

[0227] Typically, the inverse Fourier transform is an integral that can be integrated over the function g with respect to its variable values. In this case, the inverse Fourier transform can be represented by the following equation:

[0228]

[0229] To recover the discrete-time data sequence x[n], an Inverse Discrete Fourier Transform (IDFT) may be used.

[0230]

[0231] Using the Inverse Fast Fourier Transform (IFFT), the equation becomes

[0232]

[0233] Wherein a sequence f(n) of N samples is indexed from n=0 to N-1, and the discrete Fourier transform (DFT) is defined as F(k), where k=0 to N-1.

[0234]

[0235] Figures 17A to 17B An example of a three-dimensional cepstrum waveform signal 1700 is described. Waveform signal 1700 is the result of performing cepstrum analysis on a micro-motion signal, resulting in cepstrum signal waveforms 1701, 1702, 1703, with each dimension of the micro-motion signal having a cepstrum signal waveform. Cepstrum waveform signal 1700 has different features associated with it and can be easily used to identify a user. For example, the first N peaks of cepstrum waveform signal 1700 can be a predetermined feature used to extract from each cepstrum waveform signal 1700 to identify a user. The peaks are selected because of their high variation, so they are different features. For each user, the values ​​of the cepstrum and amplitude of the first N peaks can be used as the NFP of the corresponding user. The values ​​of the cepstrum and amplitude of the N peaks can be used as the NFP that is input into the classifier model in response to the authorized user calibration parameters to generate a match percentage (MP).

[0236] Other features may be selected and their values ​​extracted from the cepstrum of the micromotion signal. Other signal processing may be performed so that additional features or patterns in the micromotion signal become apparent and can be used to generate the NFP. For the NFP certification system, consistency in which features are used to extract values ​​is critical. The features should be predetermined. The same features (e.g., the top N peaks) should be used in conjunction with the calibration NFP during calibration / training mode and to form the regenerated NFP from a new sample set when in user mode.

[0237] For example, in Fig.17A As can be seen in FIG. 1 , the first five peaks P1, P2, P3, P4, P5 at their corresponding quefrequencies (or quefrencies) and amplitudes on each axis (e.g., from 3 axes / 3 curves and 5 peaks, a total of 3×5 amplitude values) have a large variation that can be used to identify users differently. The amplitudes and inverse frequencies of the first five peaks P1, P2, P3, P4, P5 will be significantly different when captured from different users. Therefore, the amplitudes and inverse frequencies extracted for the first five peaks P1, P2, P3, P4, P5 can be used to distinguish user identities, for example, Figures 18A to 18B Although the first five peaks are used in this example, this is not a constraint or limitation on the embodiments, as fewer peaks, additional peaks, or a plurality of other features may be selected to form the NFP.

[0238] Where the first five peaks P1, P2, P3, P4, P5 are predetermined features used to extract values ​​from the cepstrum signal waveform, the NFP can be repeatedly reproduced for the same user with minor variations. If a different user generates his / her NFP, the extracted values ​​of the predetermined features will vary substantially. Substantial variations between the NFP of an authorized user and the NFP of a different user can be used to grant access to the authorized user and deny access to the different user.

[0239] The first N features extracted from the cepstrum waveform can be used to regenerate the NFP and used by the NFP certification classifier to classify matches with authorized user calibration parameters. Selecting the first features in the cepstrum, such as the first N (e.g., N is 3 to 5) peaks or other coefficients, can reduce the sampling period, but still represent most of the changes in the signal. For example, these first N coefficients can be expanded to 128 extracted features by projecting N features via PCA analysis, and thus become a linear combination of the 128 extracted features.

[0240] The values ​​extracted from the cepstrum waveform are the position of the predetermined feature (the cepstrum) and its corresponding amplitude within a sampling period of 5, 10 or 20 seconds. If the predetermined feature is, for example, the first five peaks, then the values ​​of the amplitude and cepstrum for each axis are extracted as the NFP. This method summarizes the time within the sampling period.

[0241] Instead of summing over time within the sampling period, the cepstrum can be analyzed in near real time. For example, in this case, a peak at a cepstrum can be selected and plotted over time. Fig.17A The peak value P1 of the cepstrum 8 in the image is plotted. The 3D coordinates of each axis amplitude at the cepstrum 8 can then be plotted against time. Other features at other cepstrum can be further selected so there are N frequencies (e.g. 128 'cepstrum') and thus an n-space analysis can be performed using nearest neighbor analysis, linear or quadratic analysis where n is 128 for example.

[0242] However, the analysis of 128 inverse frequencies is a challenge. The analysis can be made simpler and computation time reduced by examining where the largest changes are in the waveform. The first 3, 5, or 10 quefrequencies (or quefrencies) locations can be taken and a principal component analysis (PCA) performed in 3, 5, or 10 dimensions of the data space. Then, a nearest neighbor analysis can be performed on this restricted data space, for example, to extract the NFP value.

[0243] Another way to extract features and generate NFP values ​​can be done without 3D plotting the data points for each cetfrequency according to time. The cetfrequency, amplitude, and time can be treated as true coordinates of a vector. N cetfrequency can be selected to plot amplitude and time as dimensions. These vectors can then be plotted for the resolved frequencies at any given time. This produces a time series of the extracted features that can be analyzed to extract the NFP value.

[0244] Another way to feature extract and generate NFP values ​​is to extract the frequency (inverse frequency), amplitude at different times (such as 50 times per second reminder: between 16 and 30+Hz, the original signal of interest and relevance is bandpass). In a few seconds, a series of numbers are generated that can be used as extracted values ​​of NFP. Values ​​at different time points instead of different inverse frequencies can be extracted and used as NFP.

[0245] Although cepstrum signal processing is described herein, other signal processing algorithms may be used on the micro-motion signal so that other patterns appear in different waveforms. Different features may be extracted from the waveform signal and used to define NFPs for the NFP certification classifier. Different extracted features may be used as NFPs that are input into the model in response to authorized user calibration parameters to generate a match percentage (MP).

[0246] Reference Figures 18A to 18B and 19A-19B, showing examples of a single axis of different NFPs for different users. Additional dimensionality may be provided by two or more additional dimensions.

[0247] Figures 18A to 18B An example of different NFPs based on a single axis of inverse frequency for two different users is shown. Figures 18A to 18B In each of , one or more of the five extracted peaks P1 to P5 appear with different amplitudes at different inverse frequencies F1 to F5. Therefore, the NFP of each user is different. Figures 19A to 19B An example of different NFPs based on a single axis of time for two different users. Figures 19A to 19B In each of the five peaks P1 to P5, one or more of the five peaks P1 to P5 appear at different times T1 to T5 with different amplitudes. Therefore, the NFP of each user is different.

[0248] The first five peaks of the cepstrum signal waveform are predetermined features extracted from each user's different micro-motion signals for each user. The values ​​of the positioning and amplitude of the first five peaks distinguish one user from another, similar to how the teeth and notches in a door key distinguish the key. The authorized user calibration parameters behave a bit like the tumbler of a lock that engages the teeth and notches in the door key. If the teeth and notches in the door key are wrong, the tumbler of the door lock will not engage correctly and will not open the door. In the case where more than one dimension (e.g., 3 axes) is used for feature extraction, the NFP becomes more unique. Fig. 14B An added dimension AD 1460A illustrates the value of features extracted in addition to the three dimensions of features extracted from the micro-motion signals 1460X, 1460Y, 1460Z to form the NFP 1460.

[0249] Although 3D acceleration or 3D displacement values ​​may be used, time may be another added dimension. At one point in time, the 3D value will have one amplitude, and at another point in time, it will have another amplitude.

[0250] Training / calibration

[0251] Reference Fig. 14B , one or more dimensions in NFP 1460 and user calibration parameters (if trained) are coupled to NFP certification classifier 1402. The algorithm used for the NFP certification classifier is a data mining algorithm, such as a linear regression algorithm, a nonlinear regression algorithm, a linear quadratic algorithm, a quadratic equation algorithm, a nearest neighbor classifier, or a k×N nearest neighbor classifier. The linear regression algorithm is shown as a row matrix 1452 with xM elements or xM columns multiplied by a column matrix 1454 with xM rows. A row matrix 1452 represents the NFP 1460 with the value of the extracted features for each of the one or more dimensions. A column matrix 1454 represents the authorized user calibration parameters 1466. In this case, the multiplication provides a value for the matching percentage. During training / calibration, the user calibration parameters 1466 are adjusted to form the calibration level as a matching percentage (MP) output 1465.

[0252] The NFP certified classifier 1402 is trained / calibrated by an authorized user prior to use. Once the initial training / calibration is complete, unauthorized users cannot train or recalibrate the classifier 1402. Micro-motion signals are captured by the accelerometer during one or more sampling periods, where the electronic device is in the user's hand or the user presses the touch sensor. Calibration of the accelerometer can occur with the left hand or the right hand or both hands. Calibration of the touch sensor can occur with one or more fingers of each hand.

[0253] The micro-motion signal is processed using a signal processing algorithm and features are extracted from the synthetic waveform as NFP calibration samples, referred to herein as calibration NFP. During training / calibration, the NFP calibration samples are used to find the authorized user calibration parameters 1466 associated with the authorized user and train / calibrate the classifier to produce the desired calibration match percentage level. After the authorized user calibration parameters 1466 have been generated and securely stored, the calibration NFP is abandoned for security reasons. Since the calibration NFP is not saved, it is necessary to regenerate the NFP (which is referred to as the regenerated NFP) for the user in user mode to gain access to the electronic device. After the device authentication has been re-enacted to lock the device from being accessed, the NFP needs to be regenerated by the authorized user.

[0254] The regenerated NFP is generated using another sample of neuromechanical micro-movements from the user and by re-calling the stored authorized user calibration parameters from the storage device. The regenerated NFP is then used for authentication purposes to obtain access to the electronic device. Using the regenerated NFP, the authorized user calibration parameters may provide a high match percentage so that access to the electronic device may be granted again. As the calibration NFP is abandoned, the regenerated NFP is not saved after being temporarily used to measure authentication. The regenerated NFP is then abandoned for safety reasons, and the next regenerated NFP is generated and evaluated for its match percentage with the authorized user calibration parameters. If the regenerated NFP is classified as having a match percentage greater than or equal to the access match level, access to the electronic device is granted.

[0255] Depending on the algorithm used by the classifier 1402, the training of the NFP certification classifier may be linear or non-linear (ie, non-linear).

[0256] To calibrate the classifier 1402, a predetermined value is selected for the calibration match percentage level. The predetermined value of the calibration match percentage level may be 100% or less. However, the predetermined value of the calibration match percentage level should be higher than the desired granted access match (AM) level.

[0257] In view of the NFP calibration sample, the processor in the electronic device goes through a test sequence of user calibration parameters in order to produce a match percentage equal to or greater than a predetermined value of the calibration match percentage level. Before selecting the initial user calibration parameters that produce a match percentage equal to or greater than the calibration match percentage level, the user may be asked to produce additional micro-movements (representing NFPs) by continuing to hold the device or continuing to press a touch-sensitive button. These additional NFPs are used to verify that the classifier is correctly trained / calibrated by the user calibration parameters 1466.

[0258] After calibration, experiments were conducted to determine the error rate in authorizing access to authorized users. Using cepstrum signal analysis of micro-motion signals, the trained classifier in a handheld electronic device with an accelerometer sensor had a 7% error rate in identifying authorized users. At this point, 93% of authorized users were correctly identified. Therefore, the access match (AM) level needs to be set to below 93% (e.g., 80%) so that the authorized user is always authorized to access the electronic device.

[0259] When cepstral signal analysis of micro-motion signals is used in conjunction with a touch-sensitive keypad in an electronic device, the false results appear to be improved. A 100% recognition rate of authorized users was achieved using the method of cepstral signal analysis when a small sample size of users entered their PIN numbers on the touch-sensitive keypad. However, the access match (AM) level needs to be set to less than 100% (e.g., 85%) so that the authorized user is always authorized to access the electronic device.

[0260] Cepstrum signal processing and alternative ways to generate NFP.

[0261] Many existing time domain methods for signal classification tasks are based on the existence of a fairly simple underlying pattern or template (known a priori or learned from the data). However, for real signals (such as cardiac, speech, or motor systems), such a simple pattern rarely exists due to the complexity of the underlying process. Indeed, frequency-based techniques are based on the existence of spectral patterns. From a random process point of view, frequency-based signal processing techniques used on micro-motion signals will only capture the first-order and second-order characteristics of the system.

[0262] The cepstrum analysis described in this article can extract some time-dependent information from accelerometer data, but this method is still associated with frequency dispersion. Since there is more information in the micromotion signal than ordinary frequencies, other signal processing techniques and feature extraction methods may be able to improve the strength of the NFP so that it is less sensitive and less likely to change over time.

[0263] Alternatively, chaos analysis can be used, for example, to generate NFPs. Using chaos, simulated predetermined features described on the cepstrum can be extracted. However, these predetermined features are not frequency-based, but have a time trajectory. Example features for extraction using chaos analysis are the center of gravity of a strange attractor and its fractal dimension and the Lyapunov exponent. Data scans over several seconds (e.g., 5, 10, 20 seconds) can be used to obtain the values ​​of these features from the micro-motion signal.

[0264] Another example of using chaos analysis is to use a waveform signal generated from the logarithm of the spectrum of each axis without IFT. From the logarithmic waveform signal, amplitude and time can be taken as the true coordinates of a vector and plotted for N (e.g., N=128) resolved frequencies at any given time. The center of gravity, dimension, and Lyapunov exponent can be extracted as a time series of values ​​representing the NFP.

[0265] According to one embodiment, a phase scatter or Poincare scatter may be generated from which NFP signal features may be extracted and coupled to an NFP certification classifier (see, e.g., Figures 2A to 2C 200A to 200D shown in FIG. 200B ). There is more information in the trajectory of phase scatter or Poincare scatter than in the trajectory of the cepstrum waveform. Therefore, it is useful to use the phase scatter or Poincare scatter generated from the filtered micro-motion data stream to extract features in the sensor data from the phase scatter or Poincare scatter for NFP.

[0266] In yet another embodiment, a hidden Markov model analysis is used as a signal processing module to obtain more feature information as NFP to couple into a selected classifier. A Markov model is a stochastic model used to model a stochastically changing system, in which the future state depends only on the current state and does not depend on the sequence of events before it. Typically, a Markov chain (a state space of one or more states) is used as the Markov model. The transition from one state to another is a memoryless random process. The next state depends only on the current state and does not depend on the sequence of events before it. A hidden Markov model is a Markov chain that can only partially observe states.

[0267] Reference Fig. 20 , illustrating a state machine 2000 that implements the hidden Markov model analysis. The state machine 2000 and the hidden Markov model are used to extract features from the signal processed waveform of the micro-motion signal and perform classification (or authenticate the user) used to grant or deny access.

[0268] The state machine 2000 includes states 2001-2004 and extracted features F1 2010A-Fn 2010N from the signal processed waveform (e.g., Figures 2A to 2D ), where N is less than or equal to 10. State 2001 is the accept state, state 2002 is the recalibrate state, state 2003 is the reject state, and state 2004 is the reset / initialize state.

[0269] There are state transitions that can be made between states 2001 to 2004 of the state machine 2000. State transition a1 is from the accept state 2001 to the recalibrate state 2002. State transition a2 is from the recalibrate state 2002 to the reject state 2003. State transition a3 is from the recalibrate state 2002 to the accept state 2001. State transition a4 is from the accept state 2001 to the reject state 2003. The reject state 2003 can also transition to the reset / initialize state 2004.

[0270] From state 2001 to 2003 to feature F1 2010A to Fn 2010N, there are output probabilities. From accept state 2001 to feature F1 2010A to Fn 2010N, there are output probabilities Oa1 to Oan. From recalibrate state 2002 to feature F1 2010A to Fn 2010N, there are output probabilities Or1 to Orn. ​​From reject state 2003 to feature F1 2010A to Fn 2010N, there are output probabilities Od1 to Odn.

[0271] Using a hidden Markov model, an attempt is made to get the correct answer without knowing the internal details of the system being analyzed. The answer is states 2001 to 2003, which are very complex and based on chaotic inputs. The extracted N features F1 2010A to Fn 2010N are unique measurable features. The features can be diverse, such as the center of gravity or a series of Lyapunov exponents, any parameters that can be extracted from the data trace, each of which is associated with a probability specific to three states 2001 to 2003 (e.g., Oa1 to Oan, Or1 to Orn, Od1 to Odn). The state machine 2002 is exited by a transition to a fourth state 2004. The fourth state 2004 corresponds to a system reinitialization, such as a complete reset.

[0272] In yet another embodiment, a new signal classification method is used to perform dynamic system signal analysis based on reconstructed phase space (RPS). Dynamic invariants can be used as features to be extracted after the signal analysis. It is expected that this method can capture more information, resulting in better user recognition.

[0273] How to use

[0274] like Figures 8A to 8CAs shown in FIG, a person holds an electronic device or touches one or more touch-sensitive pads to authenticate himself / herself. Within a few seconds of the sampling period, the 3D sensor (3 sensors positioned on different axes) in the electronic device senses vibrations or micro-motions in the user's hand or fingers due to the physiological conditions of the user's body. The 3D sensor simultaneously generates three electronic signals that can be sampled and converted into digital form. The digital samples are pre-processed and analyzed using signal processing techniques to generate NFPs in real time for the user who just held the electronic device or touched the touch sensor.

[0275] In response to the authorized user calibration parameters, the NFP can then be used to assess whether the person holding or touching the electronic device is an authorized user. In response to the authorized user calibration parameters, the NFP can be used as a key for encrypting and decrypting data files. Alternatively, a valid NFP can be used to authenticate whether a file is signed, written, or created by the authorized user.

[0276] The user calibration parameters may be stored in a local archive or local storage device in the electronic device. Alternatively, the user calibration parameters may be stored in a remote archive in a storage device associated with a server or in a storage device of a storage area network in the Internet cloud. Without a live authorized user's hand or finger, the authorized user calibration parameters are useless. In the event that the authorized user's hand or finger does not regenerate his / her NFP, access to the electronic device is denied.

[0277] Application for NFP certification

[0278] The NFP authentication system may be used to control access to an electronic device. The NFP authentication system may also be used to control access to functions associated with the electronic device and software applications executed by the electronic device. The NFP authentication system may be used to control access to a remote electronic device (i.e., where sensing is performed on a local device to determine access to the remote electronic device).

[0279] Examples of functions that may have access controlled by the NFP authentication system include, but are not limited to, login, user protected access, electronic transactions, and any other local functions that require explicit authentication.

[0280] For example, security applications for the NFP authentication system include computer and software login, e-commerce, electronic banking, and anti-fraud applications. Examples of human control applications for the NFP authentication system include home automation (domotic) authentication and protection (home security systems), automotive safety, and professional access to restricted areas. Examples of medical applications for the NFP authentication system include diagnostic assistance for health professionals (neuromuscular ao), treatment monitoring for patients, and patient and physician medical authentication for databases storing medical records (recording, data routing, ...). Examples of health and wellness applications for the NFP authentication system include protecting the user's physical records. Examples of gaming applications for the NFP authentication system include security features and / or simulators for virtual reality games.

[0281] Biometric encryption combines touch recognition technology with actual data storage. Data is encrypted using NFP so that it is unreadable except by the authorized user / owner of the data. Backdoor access is not available. However, other authorized users who are trusted personnel can also generate authorized user calibration parameters so that they are granted access in case of problems with the original user / owner.

[0282] Touch recognition technology can be embedded in certain applications. For example, touch recognition technology can be used for access control panels so that when the surface is touched by a human hand or a human finger, access to a vehicle, a building (e.g., a home or office), or an area (e.g., a floor) can be granted or denied in response to a user NFP without using a key (keyless). Similarly, touch recognition technology can be used for a wireless remote control key held in the user's hand or touched by a finger so that access can be granted or denied in response to a user NFP without using a key. Touch recognition technology can be combined with fingerprint authentication. In this case, an image scanner and a touch-sensitive pad can be used together to simultaneously capture both a user's fingerprint and a user's NFP for multiple authentications in granting or denying access.

[0283] Using additional signal processing algorithms, the generation of NFP can be adapted to monitor the health and fitness of the user for health correction. The change in NFP regeneration can reduce the matching percentage indicating degradation of the user's health. On the other hand, the increase in the matching percentage when regenerating NFP can indicate an improvement in the health and fitness of the user. The changes can be used to discover and / or monitor neurodegenerative diseases (such as Alzheimer's disease, Parkinson's disease) or for therapeutic effect monitoring. The changes can be used to obtain a measure of heart rate variability, and an accelerometer can be used to measure the association of stress and emotional state by extracting the shock waves of the heart pulse when the user holds the device. The changes can be used as a physiological safety feature of virtual reality glasses and goggles that can be used, for example, in the video game industry.

[0284] As previously referenced Figure 6As discussed, micromotions and micromotion signals generated from the micromotions can indicate several pathological conditions. Medical devices and controllers using touch technology / accelerometers can focus on analyzing selected frequencies associated with these pathological conditions to determine whether the health condition is healthy or degraded. Medical devices using touch technology can also monitor changes in NFP regeneration over time to detect changes in health and health. Changes or drifts in NFP values ​​between two or more time periods can be used as a way to monitor the health of the user.

[0285] The change or drift of NFP values ​​can be qualified and / or quantified using one or more different analysis methods. One method to qualify and / or quantify the deviation would be to distinguish intention tremor from rest tremor. Another such method would be frequency or Fourier analysis. Yet another method would be to use a classifier based on chaos mathematics to improve diagnosis. Yet another method would be to perform a comparison of the same user at different times, such as, for example, between medication doses or between with and without medication.

[0286] Furthermore, micromotions and generated micromotion signals may vary when captured from different important parts of the body (e.g., hands, limbs, chest) and even two different sides of the body (e.g., left and right arms). Body tremors may vary when captured from different parts of the body. Monitoring body tremors in the chest may indicate a user's thyroid condition, a fever condition, or a malignant hyperthermia condition. Monitoring the chest near the heart may enable measurement of heart rate variability (HRV).

[0287] Reference Fig.21 , the figure shows a pair of medical or health diagnostic devices 2100A to 2100B coupled to communicate with a server 2102. The medical or health diagnostic devices 2100A to 2100B may be located locally on the server and communicate using a local area network (LAN). Alternatively, the medical or health diagnostic devices 2100A to 2100B may be located remotely from the server and communicate with the server via a wide area network (WAN) (e.g., the Internet). The medical or health diagnostic devices 2100A to 2100B may be held or touched by a user to sample the micro-movements of the user. Alternatively, the medical or health diagnostic devices 2100A to 2100B may be placed and / or fastened to adjacent portions of the user's body (e.g., chest) to capture the user's micro-movements at different areas and limbs of their body in order to perform different diagnoses. The medical or health diagnostic device 2100A-2100B may include a user interface device (eg, touch display 706, keypad 705, buttons 704, and / or audio speaker / microphone) and provide a user interface to instruct the user how to use it for various types of diagnostic tests.

[0288] The medical or health diagnostic device 2100A to 2100B samples the micro-motion of the user over time (different dates and times), generates and stores micro-motion signals in response thereto for diagnosing changes in the user's health. The captured micro-motion signals are stored as data DATA that varies over time using a date and time (date / time) stamp. tn The stored micro-motion signal DATA can be tn One or more of the stored data DATA performs frequency analysis to detect certain pathological conditions. tn It may also be analyzed to extract NFPs at each time stamp. A temporal analysis may be performed to compare NFPs over date / time in order to detect changes between two or more NFPs. A report illustrating the frequency analysis of stored micro-motions over date / time and the temporal analysis of NFPs may then be generated.

[0289] The medical or health diagnostic device 2100A includes a controller 2110A and a three-dimensional accelerometer 2111A, which is used to sample the micro-movements of the user at a given date and time and generate a three-dimensional signal in response thereto. The medical or health diagnostic device 2100A can communicate with a server or other computer by wire or wirelessly so that the three-dimensional signal from the medical or health diagnostic device 2100A can be uploaded to the server 2102. The accelerometer 2111A can be used to analyze the motor movements of the user's body parts for micro-movements.

[0290] The medical or health diagnosis device 2100B includes a controller 2110B and a multi-dimensional touch sensor 2111B of at least two dimensions, and the multi-dimensional touch sensor 2111B is used to sample the micro-motion of the user at a given date and time and generate a multi-dimensional (at least two-dimensional) micro-motion signal in response thereto. The medical or health diagnosis device 2100B can communicate with a server or other computer by wire or wirelessly so that the multi-dimensional signal can be uploaded to the server 2102.

[0291] In addition to one or more processors, the server 2102 also includes a clock 2112 and a storage device 2114. The clock 2112 generates a date and time stamp that can be attached to data representing multi-dimensional micro-motion signals generated at a given time and date. The storage device 2114 stores a plurality of data DATA representing multi-dimensional micro-motion signals captured over time associated with a user / patient. t1 to DATA tn .

[0292] The server 2102 may further include a classifier 2104, a comparator 2106, and a report generator 2108. The classifier 2104, the comparator 2106, and the report generator 2108 may be a processor (e.g., Figure 7 Processor 701 in.

[0293] The classifier 2104 may be a processor configured to perform one or more of the functional processes of the NFP authentication controller. The classifier 2104 selects the plurality of data DATA stored in the storage device 2114 from the plurality of data DATA stored in the storage device 2114. t1 to DATA tn Generate NFP. That is, after storing data DATA tn Analyzed to extract NFP at each timestamp tn .

[0294] Comparator 2106 compares at least two NFPs (NFPi and NFPj) in different dates / times to detect changes between the two. If there is a negative difference between them that is greater than a predetermined percentage over time, it may indicate a degradation of the user's health. If there is a positive difference between them that is greater than a predetermined percentage over time, it may indicate an improvement in the user's health. The predetermined percentage may be set in response to an error tolerance (e.g., 5%) of the system measuring the NFP.

[0295] The report generator 2108 generates a report illustrating the frequency analysis of the stored micro-movements and the time analysis of the NFP as a function of date / time. If a healthy correction is detected, the change is reported to the user. If the number is within the predetermined percentage, an indication that no healthy correction was detected is reported to the user.

[0296] While the comparator 2106 analyzes multiple neural mechanical fingerprints that vary over time, additional analysis may be performed by the devices and servers of the system. The system may further analyze the frequency components in each of the multiple multi-dimensional motion signals to attempt to determine the pathological state based on the frequency. In addition, the system may further search for one or more predetermined data patterns in the multi-dimensional motion signals and / or neural mechanical fingerprints to determine whether the pathological state has a common data pattern or signature.

[0297] In addition to diagnosing the user's health, NFP technology can also be used to protect medical records through better authentication of patients and doctors and encryption of medical records. The user can authenticate his / her interactions with the healthcare system (providers, insurance companies, medical record companies, etc.). The user can store his / her medical records securely in the cloud or on any device in an encrypted format, where NFP is the key.

[0298] NFP technology can be used for online services. Governments can provide online services (such as voting), and touch technology can be used to verify the identity of authorized users. Schools and educators can provide online tests. Touch technology can be used to verify the identity of users on a web page or website portal (the user is logged into the web page or website portal). For example, a student taking an online test, for example, can be easily identified using NFP touch technology, even if the student is taking the test at school. Electronic online banking can become more secure, where banks can use touch technology to verify the identity of their customers.

[0299] Benefits of NFP Certification

[0300] There are several advantages to using NFP for touch recognition. Touch recognition does not require a centralized database. Access to the electronic device is locally controlled and shielded by the local NFP user authentication system.

[0301] An additional advantage of NFP is that a person can be uniquely identified as the correct user of an electronic device but remain anonymous, thereby providing strong user privacy protection. NFP is generated in response to brain / nervous system related signals generated by sensors. NFP can then provide foolproof unique user identification (even if twins have different muscle strength).

[0302] Touch recognition employs a neural algorithm to generate NFPs from micro-motion signals associated with micro-motions of fingers on the hand. This avoids the context-dependent or motion spectrum associated with behavioral biometrics. Touch recognition is user-friendly and can provide frictionless login and authentication.

[0303] The neural algorithm can be developed as software and added to previously existing electronic devices with an application programming interface (API) or a mobile software development kit. It can be added as part of the system login.

[0304] The neural algorithm does not need to continuously monitor the sensor. The neural algorithm can be in a sleep state until the electronic device is awakened from the sleep state or the sensor senses a touch. The training time and recognition are fast (only a few seconds). Therefore, the neural algorithm can save power and intelligently use the battery power that is usually available for mobile device electronic devices.

[0305] in conclusion

[0306] When implemented in software, the elements of the embodiments are essentially code segments that can be executed by one or more processors to perform and implement tasks and provide functions. For example, a processor (e.g., Figure 7The processor 701 in the embodiment may be configured by instructions (in hardware, software, or a combination of hardware and software) to perform the functional processes of the NFP authentication controller described herein. The program instructions or program code segments may be stored in a processor-readable medium or storage device coupled to one or more processors or at least in communication with one or more processors (e.g., Figure 7 The processor-readable medium may include a storage device or any medium that can store information. Examples of processor-readable media include, but are not limited to, electronic circuits, semiconductor memory devices, read-only memory (ROM), flash memory, erasable programmable read-only memory (EPROM), floppy disks, CD-ROMs, optical disks, hard disks, or solid-state drives. Programs or code segments may be downloaded or transmitted, for example, between storage devices over a computer network (e.g., the Internet, an intranet, etc.).

[0307] Although this specification contains many details, these details should not be interpreted as limitations on the scope of the invention or what may be claimed, but rather, should be interpreted as descriptions of features specific to a particular embodiment of the invention. Certain features described in the context of a separate embodiment in the present invention may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in sub-combinations in multiple embodiments. In addition, although features may be described above as working in certain combinations and even initially claimed, in some cases, one or more features from the claimed combination may be deleted from the combination, and the claimed combination may involve a sub-combination or a variation of the sub-combination.

[0308] Thus, although certain exemplary embodiments have been particularly described and shown in the accompanying drawings, they should not be construed as being limited to these embodiments, but instead should be construed in accordance with the appended claims.

Claims

1. A device for determining a change in a user's health, the device comprising: processor; as well as a memory coupled to the processor and storing instructions, wherein the instructions are executed by the processor to cause the device to: a) sensing multi-dimensional motion of a body part of the user at a first time and date to generate a first multi-dimensional motion signal; b) generating a first neuromechanical fingerprint based on the first multi-dimensional motion signal by extracting a plurality of predetermined features associated with the first multi-dimensional motion signal, wherein the first neuromechanical fingerprint is a signal associated with neuromuscular micro-movements of the user; c) generating a first metric based on the first neural mechanical fingerprint; d) sensing the multi-dimensional motion of the body part of the user at another time and date for consistency to generate another multi-dimensional motion signal; e) generating another neuromechanical fingerprint based on the another multi-dimensional motion signal by extracting a plurality of predetermined features associated with the another multi-dimensional motion signal, wherein the another neuromechanical fingerprint is a signal associated with the neuromuscular micro-movements of the user; f) generating another metric in response to the another neural mechanical fingerprint; and g) comparing the first metric to the further metric to determine a difference indicative of the change in the health of the user.

2. The device of claim 1 , wherein the instructions, when executed by the processor, further cause the device to: storing the first metric in association with the first time and date; and The another metric is stored in association with the another time and date.

3. The device of claim 1 , wherein the instructions, when executed by the processor, further cause the device to: Steps d through g are repeated to continue determining the difference representing a change in the health of the user.

4. The device according to claim 3, wherein The difference between the first metric and the further metric is positive.

5. The device according to claim 3, wherein The difference between the first metric and the further metric is negative.

6. The device according to claim 5, wherein A negative said difference is associated with drinking; and Execution of the instructions by the processor further causes the device to restrict the user from operating a machine or vehicle.

7. The device according to claim 5, wherein A negative difference is associated with the user's heart rate variability.

8. The device according to claim 5, wherein A negative difference is associated with a neurodegenerative disease in the user.

9. The device according to claim 8, wherein The neurodegenerative disease is Alzheimer's disease or Parkinson's disease.

10. The device of claim 1, wherein the instructions, when executed by the processor, cause the device to: locally sensing the multi-dimensional motion of the body part of the user at a first time and date to generate the first multi-dimensional motion signal; The first multi-dimensional motion signal is remotely transmitted to a server via a communication system to store the first multi-dimensional motion signal and generate a first neural mechanical fingerprint in response thereto.

11. The device of claim 1 , wherein the instructions, when executed by the processor, cause the device to: The multi-dimensional motion of the body part of the user is three-dimensionally sensed using a three-dimensional touch sensor at the first time and date to generate a three-dimensional motion signal.

12. The device of claim 1, wherein the instructions, when executed by the processor, cause the device to: The multi-dimensional motion of the body part of the user is three-dimensionally sensed using a three-dimensional accelerometer at the first time and date to generate a three-dimensional motion signal.

13. An apparatus for determining a change in a user's health, the apparatus comprising: processor; as well as a memory coupled to the processor and storing instructions, wherein the instructions are executed by the processor to cause the device to perform: sensing multi-dimensional movement of a body part using a multi-dimensional sensor at multiple dates / times; In response to the sensing, generating a plurality of multi-dimensional motion signals; generating a plurality of neuromechanical fingerprints for the plurality of dates / times by extracting a plurality of predetermined features based on each of the plurality of multi-dimensional motion signals, wherein the plurality of neuromechanical fingerprints are signals associated with neuromuscular micro-movements of the user; comparing at least two of the plurality of neuromechanical fingerprints to determine a change in the health of the user between at least two dates / times; and The health change is reported to the user.

14. A device according to claim 13, wherein the instructions executed by the processor cause the device to analyze the multiple neural mechanical fingerprints that change over time when comparing at least two of the multiple neural mechanical fingerprints, and the instructions executed by the processor further cause the device to analyze the frequency components in each of the multiple multi-dimensional motion signals to determine the pathological state of the user.

15. The device of claim 13, wherein the instructions, when executed by the processor, further cause the device to: One or more predetermined data patterns are searched in each of the plurality of neuromechanical fingerprints to determine a pathological state of the user.

16. The device of claim 14, wherein the instructions, when executed by the processor, further cause the device to: One or more predetermined data patterns are searched in each of the plurality of neuromechanical fingerprints to determine another pathological state of the user.

17. The device of claim 13, wherein the instructions, when executed by the processor, further cause the device to: The plurality of multi-dimensional motion signals are stored in a storage device for subsequent processing.

18. The device of claim 17, wherein the instructions, when executed by the processor, further cause the device to: One or more predetermined data patterns are searched in each of the plurality of multi-dimensional motion signals to determine a pathological state of the user.

Images