Biological characteristic identity authentication method based on multi-modal sensor data

Abstract

The invention discloses a biological characteristic identity authentication method based on multi-modal sensor data, belongs to the field of behavior biological characteristic recognition and security authentication of mobile terminal equipment, and is suitable for realizing non-inductive and continuous identity authentication by means of a sensor of a terminal such as a smart phone. The method comprises the following steps of: acquiring weak vibration waveforms of a terminal and reading jitter signals of an ambient light sensor in a non-inductive and synchronous manner by utilizing a native accelerometer of the terminal and the ambient light sensor when a user clicks a touch screen without additional hardware, and inputting the weak vibration waveforms of the terminal and the reading jitter signals of the ambient light sensor into an original Pyramid-CNN and Hieradical-Transformer mixed model. Extracting, fusing and scoring user behavior characteristics locally in real time at a terminal, constructing a multi-scene behavior characteristic database, and adaptively adjusting an authentication threshold value; secondary authentication or locking is triggered when operation deviates from a legal mode, the bimodal features and the original architecture form behavior features difficult to imitate, the attack resistance is high, the user acceptance degree is high, and the method is particularly suitable for financial payment and other high-risk mobile terminal scenes and provides auxiliary safety protection after main authentication.

Description

TECHNICAL FIELD

[0001] The application discloses a biometric identity authentication method based on multi-modal sensor data, belongs to the field of mobile terminal behavior biometric recognition and security authentication, and is suitable for realizing non-invasive and continuous identity verification by relying on sensors of a terminal such as a smart phone. Without additional hardware, the original accelerometer and ambient light sensor of the terminal are used to synchronously collect weak vibration waveforms of the terminal and ambient light sensor reading jitter signals when a user clicks the touch screen, and a unique Pyramidal-CNN and Hierarchical-Transformer hybrid model is input. User behavior features are extracted and fused in real time at the terminal, and a multi-scene behavior feature database is constructed to adaptively adjust the authentication threshold; secondary authentication or locking is triggered when the operation deviates from the legal mode, the dual-mode features and unique architecture form difficult-to-imitate behavior features, the anti-attack capability is strong, the user acceptance is high, and the method is particularly suitable for high-risk mobile terminal scenes such as financial payment, and provides auxiliary security protection after primary authentication. BACKGROUND

[0002] With the rapid development of mobile Internet and financial technology, intelligent terminals have become the core carriers for users to store sensitive information and perform critical operations. Identity authentication is the first line of defense for information security. However, there is a common security blind spot: once a device is unlocked through initial authentication (such as entering a password, facial recognition, or fingerprint recognition), all subsequent operations are assumed to be performed by the legitimate user. This leads to a situation where if a user's phone is lost or obtained by a familiar person, an attacker can directly access all application data and cause irreparable property and privacy loss if they obtain the unlock password or gain control of the device when it is not locked. The existing authentication system lacks a continuous and hidden secondary verification mechanism after the first verification, and cannot address the security threats after authorization.

[0003] Current mainstream identity authentication technologies can be divided into three categories, each with significant defects, making it difficult to build the above-mentioned continuous and effective security line: one is knowledge factor authentication (such as passwords, pattern locks, and dynamic verification codes), which relies on user memory or manual input and is vulnerable to shoulder surfing and phishing attacks; two is physiological biometric factor authentication (such as fingerprints and facial recognition), which is strong in uniqueness but requires specialized hardware support and is vulnerable to fake fingerprint films and synthetic photo attacks; three is existing behavior factor authentication, such as gesture authentication based on accelerometers, which requires users to actively perform specific actions and disrupts the normal use process, resulting in poor user experience. Existing methods are completely ineffective once attacked in the "authorized" scenario. These technologies have not effectively addressed the core issue: how to perform a hidden and continuous identity verification in every micro-interaction of a user's normal device use without the user's awareness or cooperation.

[0004] We have discovered that everyone exhibits unique and stable overall behavioral habits in daily life—including gait rhythm, click frequency, daily activity paths, and ambient lighting patterns. These habits constitute a "behavioral fingerprint" that is difficult to imitate. Based on this core insight, this invention uses the device's native accelerometer and ambient light sensor to capture users' micro-interaction behaviors in different typical scenarios, such as checking messages while walking, sitting down to enter passwords, and watching videos while resting. Therefore, the target scenario of this invention is: even if an attacker obtains an unlocked device, because they cannot imitate the overall behavioral habits of a legitimate user composed of subtle interactive features, their operations will be continuously identified as abnormal by the system, triggering alarms or locking. Specifically, this invention must simultaneously possess three core capabilities: first, it does not rely on the user's active cooperation, but completes authentication by capturing daily operational behaviors to achieve "registration upon use, authentication upon use"; second, it has rich feature dimensions and is physically uncopyable, effectively resisting imitation and forgery attacks; and third, it adapts to the needs of multiple usage scenarios, achieving continuous identity authentication based on the user's overall understanding of their lifestyle. Summary of the Invention

[0005] The core objective of this invention is to address the shortcomings of existing identity authentication technologies and provide a seamless, continuous mobile identity authentication solution that is highly concealed, secure, provides a superior user experience, and is adaptable to various scenarios. Specifically, it aims to: achieve seamless authentication for users, requiring no additional steps and simply capturing everyday screen taps to complete authentication without interrupting the user experience, achieving the effect of "registering and authenticating upon use"; enhance authentication security and anti-forgery capabilities by using multimodal sensor data fusion and a unique network model to construct rich, physically meaningful, and difficult-to-replicate behavioral features across different scenarios, resisting imitation and forgery attacks; and reduce hardware dependence by using only the accelerometer and ambient light sensor commonly found in mobile devices, eliminating the need for additional dedicated hardware, thus lowering device compatibility barriers and manufacturing costs.

[0006] A mobile-based, seamless, continuous identity authentication method based on multimodal behavioral biometrics is characterized by employing a unique hybrid model architecture combining Pyramidal-CNN and Hierarchical-Transformer. This method achieves highly robust identity authentication through four main steps: synchronous signal acquisition, preprocessing, multimodal feature fusion modeling, and real-time authentication. The specific steps are as follows:

[0007] (1) Synchronous signal acquisition

[0008] When a user touches the screen of a smartphone or other mobile device and performs a single-click operation (“micro-interaction”) such as clicking an icon, button, or text input, the system automatically and synchronously triggers the accelerometer and ambient light sensor through the device's underlying operating system interface to collect two types of raw signals in real time:

[0009] Vibration signal: with a sampling frequency of not less than 400Hz f v ≥400Hz, record the complete waveform generated by the accelerometer at the instant of click, acquisition time T v From the start of the covering vibration to its complete disappearance (50ms≤T) v ≤100ms), the raw data is represented as V(t), including the vibration amplitude A. v (t), frequency f v (t), decay rate Equal dimensions.

[0010] Ambient light signal: sampled at a frequency of not less than 50Hz. l ≥50Hz, capturing the change in light intensity sensed by the light sensor at the moment of click, with a collection time T. l Covering the finger from 10ms before it approaches to 10ms after it leaves (T l =t 离开后10ms -t 靠近前10ms The raw data is represented as L(t), which includes the real-time light intensity value I. l (t), rate of change Equal dimensions.

[0011] (2) Signal preprocessing

[0012] Preprocessing:

[0013] 1. Noise Reduction: The vibration signal is filtered using a Kalman filter to eliminate environmental noise. The filtered signal V'(t) satisfies the state update equation: in Let K be the filtered state at time k (vibration amplitude / frequency). k For Kalman gain, z k Let L be the original observation value at time k, and H be the observation matrix; the optical signal is filtered using a moving average, and the filtered signal L'(t) is: Where N = 5 is the size of the sliding window. The sampling interval is denoted as .

[0014] 2. Normalization: Min-max normalization unifies the signal values ​​to the [0,1] interval. The normalization formula is: Where S(t) is the original signal after filtering (V'(t) or L'(t)), and min(S) and max(S) are the minimum and maximum values ​​of the signal, respectively.

[0015] Multi-scale local feature extraction (Pyramidal-CNN module):

[0016] The preprocessed vibration signal and ambient light signal are directly input into a deep learning neural network. Since the two types of signals have different sampling rates (vibration ≥ 400Hz, light ≥ 50Hz) and heterogeneous physical meanings, this invention does not enforce strict time alignment, but rather adaptively learns their intrinsic correlation through the hierarchical structure of the model. To overcome the shortcomings of standard one-dimensional convolutional networks, such as a single receptive field and difficulty in simultaneously capturing transient pulses and long-term trends in vibration and light signals, this invention proposes a Pyramidal-CNN module. This module employs a multi-branch parallel structure, and for any input modality's normalized time series... (L is the sequence length), and four different sizes of convolutional kernels are used for processing. The operation is defined as follows: Branch 1 (Pointwise Convolution, 1×1): Uses a one-dimensional convolution with a kernel size of 1, combined with the ReLU activation function, focusing on extracting microscopic local features and transient abrupt changes in the signal. Its output is: h1 = ReLU(Conv1D) 1×1 (x)) Branch 2 (Narrow-field convolution, 3×3): Uses a one-dimensional convolution with a kernel size of 3 to capture medium-range contextual features and undulating textures of the signal. Its output is: h2 = ReLU(Conv1D) 3×3 (x)) Branch 3 (Wide-field Convolution, 5×5): Uses a one-dimensional convolution with a kernel size of 5 to obtain a more macroscopic signal trend and overall morphological envelope. Its output is: h3 = ReLU(Conv1D) 5×5 (x)) Branch 4 (Pooling and Projection): First, downsampling is performed using a max-pooling layer with a kernel size of 3 to expand the receptive field and enhance robustness. Then, a 1×1 convolution is used for nonlinear transformation and feature compression. Its output is: h4=Conv1D 1×1 (MaxPool1D 3×3 (x)) The outputs of each branch are concatenated along the channel dimension to form a multi-scale fused feature H = [h1; h2; h3; h4]. H is then trained stably using a batch normalization layer and further trained using a one-dimensional convolutional layer Conv1D with a kernel size of 1. 1×1 Cross-channel feature fusion and dimensionality reduction are performed, ultimately outputting a refined local feature vector F of fixed dimension D. p Vibration and optical signals are processed through this module to obtain their feature vectors. and (where D is the feature dimension).

[0017] Hierarchical global relational modeling (Hierarchical-Transformer module):

[0018] To overcome the attention dispersion and modal interference problems inherent in the standard Transformer when processing heterogeneous dual-modal signals, this invention proposes a Hierarchical-Transformer module. This module employs a two-level attention mechanism for structured information fusion:

[0019] First stage: Intramodal self-attention. This involves processing the feature vector output by Pyramidal-CNN. and Construct into a sequence At this stage, modality-specific position coding is introduced. With learnable modal embedding vectors The input representation is enhanced: Subsequently, self-attention within a modality is forced to be computed through a grouped attention mask. Specifically, when computing scaled dot product attention, the attention weight matrix A is calculated by the following formula: in, These are the query, key, and value matrix, respectively, W. * For the learnable parameter matrix, d k Let M be the dimension of the key vector. The mask M ensures that vibrational features interact with themselves only, and the same applies to optical features. This step aims to first enhance the intrinsic patterns and key features within each mode. The output of this stage is denoted as Z. intra .

[0020] Level 2: Intermodal cross-attention. The feature Z output by intramodal self-attention. intra Based on this, cross-modal attention is calculated. Specifically, Z... intra Disassembled into vibrational components z v And light part z l Using vibration features as the query vector. Using light features as key vectors Sum value vector Calculate the cross-attention output once: Meanwhile, light features are used as query vectors. Using vibrational characteristics as a key Sum Calculate the output of another cross-attention step: This initiative aims to explicitly and purposefully uncover the deep synergistic relationships between vibration and optical signals.

[0021] The output O of two-level attention v←l and O l←v After residual connections and layer normalization (LayerNorm), the sequences are concatenated and fed into a feed-forward network (FFN) for final nonlinear transformation and feature mapping. This FFN typically consists of two linear layers and a GELU activation function: FFN(z) = Linear2(GELU(Linear1(z))). Finally, by directly concatenating and linearly projecting the sequences output by the FFN, a highly discriminative 32-dimensional fused feature vector integrating local details and global correlations is output. As the final representation of user behavior characteristics.

[0022] Model training, threshold determination, and incremental learning:

[0023] The model training in this invention employs a supervised end-to-end learning approach. The 32-dimensional fused feature vector output by the Hierarchical-Transformer module is used. As input, it is mapped to the categorical logical value z = W through a fully connected layer. c f+b c Then, the cross-entropy loss function is used for optimization: Where B is the batch size, C is the number of user categories, and y i The data represents the true labels for the samples. Training uses a dataset containing 100-200 valid click samples from each user, split into training and test sets in an 8:2 ratio. After model training, the authentication threshold θ is determined by calculating the cosine similarity between the test set samples and the user feature template. Five-fold cross-validation is used to optimize the balance between FAR and FRR. The final model parameters are stored in the terminal's local secure area after being encrypted with AES-256.

[0024] The system employs an incremental learning mechanism to adapt to changes in user behavior. Every 100 valid operations, the system adds new samples to the learning buffer. When the buffer reaches its preset capacity, the system fine-tunes the model at a lower learning rate during idle periods. After completion, it updates the locally stored model parameters and user feature templates to ensure continuous authentication accuracy. Attached Figure Description

[0025] Figure 1 It is a biometric authentication method based on multimodal sensor data.

[0026] Figure 2 This is a schematic diagram of the dual-modal signal acquisition and preprocessing process for a biometric authentication method based on multimodal sensor data.

[0027] Figure 3 This is the core model architecture diagram of a biometric authentication method based on multimodal sensor data. Detailed Implementation

[0028] The present invention will be further described below with reference to the accompanying drawings.

[0029] Its core innovation lies in leveraging the accelerometers and ambient light sensors built into smartphones and other mobile devices to construct a seamless and continuous identity authentication system through two stages: "user registration" and "real-time authentication." During the registration stage, the system builds a template of the user's behavioral characteristics in a specific scenario, and then performs real-time comparison during the authentication stage to verify identity. This solution requires no additional hardware and can seamlessly adapt to various sensitive scenarios such as financial payments and access to private data, ensuring high security while maximizing the preservation of a natural user experience. The specific process is as follows:

[0030] (I) User Registration Stage

[0031] Step R1: As Figure Two As shown, when a user activates the system, the system will establish a database for that usage scenario, recording the user's click operations within that scenario. Scenario types include checking messages while walking, sitting down to enter a password, and watching videos while resting. Operation types cover the user's daily habits, such as password input, button operations, and text input clicks. During this period, the system simultaneously triggers the accelerometer and ambient light sensor, collecting the dual-modal raw signal V corresponding to each click according to the same specifications as the real-time authentication phase. i (t) and L i (t), where i = 1, 2, ..., N.

[0032] Step R2: As Figure Three As shown, for the N sets of collected bimodal data, preprocessing identical to that in the real-time authentication stage and the forward propagation process of the Pyramidal-CNN and Hierarchical-Transformer hybrid model are performed respectively, finally obtaining N 32-dimensional high-dimensional fusion feature vectors f1, f2, ..., f N The system will use this feature vector set. Perform aggregation.

[0033] Step R3: As Figure OneAs shown, the generated feature templates will be automatically identified and classified by the neural network, stored in a protected database scene partition on the terminal, and the threshold θ under that scene will be calculated.

[0034] (II) Real-time Authentication Phase

[0035] Step A1: As Figure Two As shown, during normal device use, each touch of the screen automatically triggers a dual-sensor synchronous data acquisition process identical to the registration phase, obtaining real-time V... i (t) and L i (t).

[0036] Step A2: As Figure Three As shown, the acquired real-time bimodal data, after preprocessing (denoising and normalization), is input into the original Pyramidal-CNN and Hierarchical-Transformer hybrid network of this invention. This ultimately generates a 32-dimensional real-time high-dimensional fused feature vector.

[0037] Step A3: As Figure One As shown, the system calculates the cosine similarity between the real-time feature vector and the locally stored user template features, which is used as the authentication confidence score S for this operation. If the single score S ≥ θ (θ is a preset threshold), it is determined to be a legitimate operation, allowing the user to continue using the service; if the single score S < θ, it is marked as a "suspicious operation". The system maintains a sliding window of length 3 to record the most recent consecutive operation results. If the number of "suspicious operations" in the window reaches 3, the system determines that there is a potential attack risk and immediately triggers security protection mechanisms (such as forcibly locking the screen, pausing ongoing sensitive transactions, sending alarm SMS messages to preset security contacts, etc.), thereby achieving seamless and continuous auxiliary identity authentication and real-time risk interception.

[0038] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the invention. Any person skilled in the art can make many possible variations and modifications to the technical solutions of the present invention, or modify them into equivalent embodiments, without departing from the scope of the present invention. Therefore, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention, without departing from the content of the present invention, should fall within the protection scope of the present invention.

Claims

1. Patent Title: A Biometric Identity Authentication Method Based on Multimodal Sensor Data, characterized in that, The process includes a user registration phase and a real-time authentication phase, with the specific steps as follows: Step 1: Synchronously acquire signals through dual-modal signal acquisition. When the user performs a mobile screen tap operation, the system automatically triggers the device's native accelerometer and ambient light sensor to acquire the original vibration signal and light intensity signal. Step 2: Preprocess the original dual-modal signal by performing noise reduction and normalization. Step 3: Input the preprocessed dual-modal signal into the hybrid network of Pyramidal-CNN and Hierarchical-Transformer. After multi-scale local feature extraction and hierarchical global association modeling, a high-dimensional fusion feature vector is generated. The identity authentication model is trained based on the user's legitimate operation samples. The authentication confidence threshold is determined by the output distribution of the fusion feature vector. The trained model parameters and behavioral features in specific scenarios are stored in the device's local database. Step 4: Real-time acquisition and preprocessing to obtain feature vectors. Each time the user clicks the screen, the system repeats the process from Step 1 to Step 2 in real time to obtain the dual-modal signal of the current operation, inputs it into the hybrid network for inference, and outputs the real-time fused feature vector and the corresponding authentication confidence score. Step 5: Use the local hybrid model to perform inference and execute security protection. Determine the legality of the current operation based on the comparison result of the authentication confidence score and the authentication threshold. Trigger the security protection mechanism for operations that are continuously determined to be suspicious.

2. The biometric authentication method based on multimodal sensor data according to claim 1, characterized in that, In step (1), the accelerometer takes the moment the finger touches the screen as the time origin and collects the vibration signal from the start of vibration to its complete disappearance at a sampling rate of not less than 400Hz. The light intensity sensor collects the light intensity change signal of the entire cycle when the finger clicks at a sampling rate of not less than 50Hz.

3. The biometric authentication method based on multimodal sensor data according to claim 1, characterized in that, In step (2), the preprocessing includes: using Kalman filtering to eliminate environmental noise for vibration signals and using moving average filtering to eliminate interference for optical signals; and compressing the preprocessed dual-mode signal values ​​to the [0,1] interval through Min-Max normalization.

4. The biometric authentication method based on multimodal sensor data according to claim 1, characterized in that, In step 3, the Pyramidal-CNN module adopts a multi-branch parallel structure, simultaneously performing 1×1 convolution, 3×3 convolution, 5×5 convolution, and 3×3 max pooling followed by 1×1 convolution on the input signal. The outputs of each branch are concatenated in the channel dimension, then batch normalized and fused with 1×1 convolution to reduce the dimensionality, outputting a refined local feature vector of fixed dimension.

5. The biometric authentication method based on multimodal sensor data according to claim 1, characterized in that, In step 3, the Hierarchical-Transformer module adopts a two-level attention mechanism: the first level of intramodal self-attention forces vibration and optical features to perform self-attention calculations separately through grouped attention masks; the second level of intermodal cross-attention calculates cross-attention with vibration features as queries and optical features as keys, and calculates cross-attention again with optical features as queries and vibration features as keys; the two-level outputs are connected by residuals, normalized by layers, and fed-forwarded by a feedforward network to generate a 32-dimensional fused feature vector.

6. The biometric authentication method based on multimodal sensor data according to claim 1, characterized in that, In step 3, the model training uses the cross-entropy loss function, and the hybrid network is optimized end-to-end using a dataset formed by legitimate user clicks. The authentication threshold is determined on the test set using five-fold cross-validation to balance the false acceptance rate and the false rejection rate.

7. The biometric authentication method based on multimodal sensor data according to claim 1, characterized in that, In step 3, the system sets up an incremental learning mechanism. For every preset number of valid operations, new samples are added to the buffer pool. When the device is idle, the model is fine-tuned with a low learning rate, and the locally stored model parameters and user templates are updated.

8. The biometric authentication method based on multimodal sensor data according to claim 1, characterized in that, The security protection mechanism in step 5 includes at least one of pausing sensitive operations, automatically locking the device screen, and sending alarm notifications to preset contacts. The mobile device includes smartphones, etc., and the accelerometer and ambient light sensor are both built-in sensors of the device.

9. A biometric authentication system based on multimodal sensor data, characterized in that, include: The signal acquisition and preprocessing module is used to simultaneously trigger the mobile accelerometer and ambient light sensor when the user performs a screen click operation to acquire the original vibration signal and the original light intensity signal. It is also used to denoise and normalize the acquired dual-modal raw signals; The dual-modal fusion modeling module, which includes the Pyramidal-CNN submodule and the Hierarchical-Transformer submodule, is used to perform multi-scale local feature extraction and hierarchical global correlation modeling on dual-modal signals, generate high-dimensional fusion feature vectors, complete model training, and determine the authentication threshold. The registration and encrypted storage module is used to aggregate feature vectors from multiple operations during the registration phase to construct user behavior feature templates for specific scenarios, and to encrypt and store the model parameters and user features for that scenario in the device's local database. The real-time authentication module is used to generate a fused feature vector in real time when the user clicks, and to calculate the similarity score with the stored user template. The security decision module is used to determine the legality of an operation based on the comparison between the authentication confidence score and the threshold, and to trigger a security protection mechanism for continuous suspicious operations.

10. Through the collaborative work of the above-mentioned multimodal sensor data acquisition, feature fusion modeling and real-time authentication system, the present invention achieves seamless and continuous verification of mobile user identity and accurate identification of suspicious operations, improves the security and convenience of identity authentication, and provides strong protection for the execution of sensitive device operations and the protection of user privacy data.

Images