Continuous identity authentication method, device and equipment based on multi-modal characteristics

Abstract

The invention relates to the technical field of continuous authentication, and provides a continuous identity authentication method, device and equipment based on multi-modal characteristics, and the method comprises the steps: collecting the motion behavior data of an authenticated user in a preset time length through a motion sensor on intelligent equipment, and collecting the touch behavior data of the authenticated user in the preset time length through a touch screen sensor; according to the motion behavior data of the authenticated user, obtaining posture features of the intelligent equipment, and according to the touch behavior data of the authenticated user, obtaining user touch features; and inputting the posture feature and the user touch feature into an identity authentication model to obtain an identity authentication result of the user. Therefore, the accuracy of continuous identity authentication of the identity authentication model is improved, the problem that the identity authentication model cannot be flexibly applied to other applications in the prior art is solved, multi-modal feature fusion is not needed, and the problems that in the prior art, excessive dependence on behavior biological feature acquisition, processing, modeling and learning of a user is caused, the model training calculation amount is large, and the user experience is poor are solved. And the complexity is high.

Description

Technical Field

[0001] The present invention relates to the field of continuous authentication technology, and in particular to a continuous identity authentication method, apparatus and device based on multimodal features. Background Art

[0002] Continuous authentication refers to a technology that uses the user's behavioral biometrics to continuously monitor and verify the user's identity. It can continuously verify the authenticity of the user's identity throughout the process of authenticating the user's use of smart devices or services, ensuring the security of the smart devices or services throughout the entire use process.

[0003] In the prior art, methods for achieving continuous identity authentication of users when using smart devices include: continuous authentication schemes based on motion sensors, specifically, using inertial sensors such as accelerometers, gyroscopes, and magnetometers built into smart devices to collect motion data of users during daily operations, so as to model their dynamic behavior patterns and achieve continuous verification of user identities. Continuous authentication based on sliding gesture features, specifically, using the interaction between the user and the touch screen as the basis for authentication, capturing the dynamic features of the user's gesture trajectory, and extracting behavioral features with individual differences for identity discrimination. Continuous authentication schemes based on multimodal fusion, specifically, overcoming the information limitations of single-modal data, introducing a multimodal fusion strategy, and integrating modeling of user behavior biometrics to achieve continuous verification of user identities.

[0004] However, the existing technologies are usually used to achieve continuous verification of user identity in specific scenarios and cannot be flexibly applied to other application scenarios. For multimodal fusion continuous authentication solutions, they rely too much on the acquisition, processing, modeling and learning of user behavioral biometrics, and there are problems with large model training computational complexity and high complexity. Summary of the Invention

[0005] Based on this, it is necessary to provide a continuous identity authentication method, device and equipment based on multimodal features to address the above technical problems.

[0006] In a first aspect, an embodiment of the present invention provides a continuous identity authentication method based on multimodal features, which is applied to a smart device, wherein a motion sensor and a touch screen sensor are provided on the smart device, including:

[0007] Collecting the user's motion behavior data for a preset time period through the motion sensor, and collecting the user's touch behavior data for a preset time period through the touch screen sensor;

[0008] Acquire the posture characteristics of the smart device based on the authenticated user motion behavior data, and acquire the user touch characteristics based on the authenticated user touch behavior data;

[0009] Inputting the gesture feature and the user touch feature into an identity authentication model to obtain an identity authentication result of the user;

[0010] In which, the identity authentication model is obtained by training the user's positive sample data, negative sample data and triple loss function, the positive sample data includes: posture features, user touch features and labels corresponding to the user, and the negative sample data includes: posture features, user touch features and labels corresponding to non-users; the identity authentication model includes: a feature extraction module, a distance measurement module and a decision classification module, the feature extraction module is used to extract the identity features of the authenticated user, the distance measurement module is used to calculate the similarity between the identity features and the positive sample data and negative sample data, and the decision classification module is used to obtain the identity authentication result based on the similarity.

[0011] In one embodiment, the authenticated user motion behavior data includes linear acceleration data, angular velocity data, and magnetic field data. Acquiring the posture features of the smart device based on the authenticated user motion behavior data includes:

[0012] determining an error correction vector according to the linear acceleration data, the magnetic field data, and a preset quaternion;

[0013] Obtaining a quaternion change rate of the preset quaternion within a preset time period according to the angular velocity data and the preset quaternion;

[0014] Calculating a partial derivative of the error correction vector to obtain an error correction Jacobian matrix corresponding to the error correction vector;

[0015] Update the preset quaternion according to the quaternion change rate, the error correction Jacobian matrix, and the error correction vector to obtain a target quaternion;

[0016] The posture feature is acquired according to the target quaternion, the linear acceleration data, the angular velocity data, and the magnetic field data.

[0017] In one embodiment, before determining the error correction vector according to the linear acceleration data, the magnetic field data, and the preset quaternion, the method further includes:

[0018] Initializing the preset quaternion;

[0019] Normalization processing is performed on the linear acceleration data and the magnetic field data.

[0020] In one embodiment, determining the error correction vector according to the linear acceleration data, the magnetic field data, and a preset quaternion includes:

[0021] Acquire first constraint data according to the preset quaternion and the magnetic field data, wherein the first constraint data is used to constrain the magnetic field data;

[0022] An error correction vector is determined according to the linear acceleration data, the magnetic field data, the first constraint data, and the preset quaternion.

[0023] In one embodiment, updating the preset quaternion according to the quaternion change rate, the error correction Jacobian matrix, and the error correction vector to obtain a target quaternion includes:

[0024] Determining a quaternion update gradient based on the error correction Jacobian matrix and the error correction vector;

[0025] Determining a quaternion update formula according to the quaternion update gradient and the quaternion change rate;

[0026] The preset quaternion is updated according to the quaternion update formula to obtain a target quaternion.

[0027] In one embodiment, acquiring the posture feature according to the target quaternion, the linear acceleration data, the angular velocity data, and the magnetic field data includes:

[0028] Obtaining the Euler angle of the posture of the smart device according to the target quaternion;

[0029] Obtaining a total acceleration and a total angular velocity respectively according to the linear acceleration data and the angular velocity data;

[0030] Frequency domain attitude features and time domain attitude features are acquired according to the linear acceleration data, the angular velocity data, the magnetic field data, the total acceleration, the total angular velocity, and the attitude Euler angle.

[0031] In one embodiment, obtaining the user touch feature according to the authenticated user touch behavior data includes:

[0032] Determine touch information corresponding to the plurality of user touch points of the preset duration, wherein the touch information includes: touch position coordinates, touch major axis, touch minor axis, and timestamp;

[0033] According to the touch information, the user touch characteristics of different preset stages within the preset time length are obtained, wherein the preset stages include: a touch sliding start stage, a touch sliding stabilization stage, and a touch sliding end stage, and the user touch characteristics include at least: position characteristics, duration characteristics, length characteristics, direction characteristics, curvature characteristics, speed characteristics and touch area characteristics.

[0034] In one embodiment, inputting the gesture feature and the user touch feature into an identity authentication model to obtain the user's identity authentication result includes:

[0035] Inputting the posture feature and the user touch feature into a feature extraction module, and extracting the identity feature of the authenticated user through the feature extraction module;

[0036] Inputting the identity feature into the distance measurement module, and calculating a first similarity between the identity feature and the positive sample data, and a second similarity between the identity feature and the negative sample data by the distance measurement module;

[0037] The first similarity and the second similarity are input into the decision classification module, and the decision classification module performs classification to obtain the user's identity authentication result.

[0038] In a second aspect, an embodiment of the present invention provides a continuous identity authentication device based on multimodal features, which is applied to a smart device. The smart device is provided with a motion sensor and a touch screen sensor, including:

[0039] A behavior data acquisition module, configured to collect the user's motion behavior data within a preset time period through the motion sensor, and collect the user's touch behavior data within a preset time period through the touch screen sensor;

[0040] a feature acquisition module, configured to acquire the posture feature of the smart device based on the authenticated user motion behavior data, and acquire the user touch feature based on the authenticated user touch behavior data;

[0041] An identity authentication result acquisition module, configured to input the posture feature and the user touch feature into an identity authentication model to obtain the user's identity authentication result;

[0042] In which, the identity authentication model is obtained by training the user's positive sample data, negative sample data and triple loss function, the positive sample data includes: posture features, user touch features and labels corresponding to the user, and the negative sample data includes: posture features, user touch features and labels corresponding to non-users; the identity authentication model includes: a feature extraction module, a distance measurement module and a decision classification module, the feature extraction module is used to extract the identity features of the authenticated user, the distance measurement module is used to calculate the similarity between the identity features and the positive sample data and negative sample data, and the decision classification module is used to obtain the identity authentication result based on the similarity.

[0043] In a third aspect, an embodiment of the present invention provides an electronic device comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, it implements the steps of the continuous identity authentication method based on multimodal features described in the first aspect.

[0044] The technical solution provided by the embodiment of the present invention has the following advantages compared with the existing technology:

[0045] An embodiment of the present invention provides a continuous identity authentication method based on multimodal features. This method uses a motion sensor on a smart device to collect motion behavior data of the user being authenticated for a preset period of time, and a touchscreen sensor to collect touch behavior data of the user being authenticated for a preset period of time. Based on the motion behavior data, the smart device's posture features are obtained, and based on the touch behavior data, the user's touch features are obtained. The posture features and user touch features are then input into an identity authentication model to obtain the user's identity authentication result. The identity authentication model is obtained by training the user's positive sample data, negative sample data and triple loss function. The positive sample data includes: posture features, user touch features and labels corresponding to the user, and the negative sample data includes: posture features, user touch features and labels corresponding to non-users. The triple loss function is used to achieve effective clustering of user identity features, thereby improving the accuracy of the identity authentication model in performing continuous identity authentication. Moreover, it can realize continuous identity authentication of users using smart devices based on the authenticated user's motion behavior data and the authenticated user's touch behavior data, solving the problem that the existing technology cannot be flexibly applied to other applications. The identity authentication model includes: a feature extraction module for extracting the identity features of the authenticated user, a distance measurement module for calculating the similarity between the identity features and the positive sample data and the negative sample data, and a decision classification module for obtaining the identity authentication result based on the similarity. In this way, when using the identity authentication model to realize continuous identity authentication, there is no need for multimodal feature fusion, avoiding the problem of excessive reliance on the acquisition, processing, modeling and learning of the user's behavioral biometric features in the existing technology, which has large model training calculation amount and high complexity. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0047] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0048] Figure 1A flowchart of a continuous identity authentication method based on multimodal features provided by an embodiment of the present invention;

[0049] Figure 2 A schematic diagram of a directional feature provided by an embodiment of the present invention;

[0050] Figure 3 A schematic diagram of the speeds of different touch points during three sliding processes of the same sliding gesture within a preset time T provided by an embodiment of the present invention;

[0051] Figure 4 A schematic diagram of touch areas of different touch points during three sliding movements of the same sliding gesture within a preset time T provided by an embodiment of the present invention;

[0052] Figure 5 A schematic diagram of a network model of an identity authentication model provided by an embodiment of the present invention;

[0053] Figure 6 A schematic diagram of the structure of a continuous identity authentication device based on multimodal features provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0054] In order to more clearly understand the above-mentioned objects, features and advantages of the present invention, the solutions of the present invention will be further described below. It should be noted that, in the absence of conflict, the embodiments of the present invention and the features therein can be combined with each other.

[0055] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present invention, rather than all the embodiments.

[0056] In one embodiment, Figure 1 As shown, Figure 1 This is a flow chart of a continuous identity authentication method based on multimodal features provided by an embodiment of the present invention. The present invention is applied to a smart device. A motion sensor and a touch screen sensor are provided on the smart device. The motion sensor and the touch screen sensor are used to collect sensor data when the user uses the smart device. The method specifically includes the following steps:

[0057] S10: Collecting the user's motion behavior data within a preset time period through the motion sensor, and collecting the user's touch behavior data within a preset time period through the touch screen sensor.

[0058] Among them, the preset duration refers to the duration of time for collecting sensor data of the user in the process of using the smart device through a motion sensor or a touch screen sensor. The preset duration can be, for example, the duration used by the user to perform a sliding gesture on the screen of the smart device, but is not limited to this. The present invention is not specifically limited thereto, and those skilled in the art can set it according to actual conditions.

[0059] The above-mentioned motion sensor includes: accelerometer, gyroscope and magnetometer. Based on this, the authenticated user motion behavior data includes: linear acceleration data, angular velocity data and magnetic field data. For example, the accelerometer, gyroscope and magnetometer included in the motion sensor respectively collect linear acceleration data corresponding to different collection moments within a preset time period. x (t),a y (t),a z (t), angular velocity data ω x (t),ω y (t),ω z (t) and magnetic field strength data m x (t),m y (t),m z (t). However, this is not limited to the present invention, and those skilled in the art can set it according to actual conditions.

[0060] The authenticated user touch behavior data includes touch position coordinates, touch major axis, and touch minor axis. For example, the touch position coordinates x(t), y(t), touch major axis l(t), and touch minor axis w(t) corresponding to different acquisition times within a preset time period are collected by the touch screen sensor. However, this is not a limitation of the present invention and can be configured by those skilled in the art based on actual circumstances.

[0061] It should be noted that the authenticated user's motion behavior data and the authenticated user's touch behavior data are mapped one-to-one based on the timestamp. Furthermore, the authenticated user's touch behavior data and the authenticated user's touch behavior data are de-noised to reduce noise introduced by the external environment during the collection process and minimize errors in the user's ongoing identity authentication.

[0062] Specifically, when a user uses a smart device, the motion sensor set on the smart device collects and authenticates the user's motion behavior data within a preset time period, and the touch screen sensor set on the smart device collects and authenticates the user's touch behavior data within a preset time period.

[0063] S11: Acquire posture features of the smart device based on the authenticated user motion behavior data, and acquire user touch features based on the authenticated user touch behavior data.

[0064] Among them, the posture features of the smart device and the user touch features can represent the user's behavioral habits when using the smart device.

[0065] Specifically, after obtaining the authenticated user motion behavior data, the posture features of the smart device are further obtained based on the authenticated user motion behavior data. After obtaining the authenticated user touch behavior data, the user touch features are further obtained based on the authenticated user touch behavior data.

[0066] Optionally, based on the above embodiment, in some embodiments of the present invention, one implementation method for obtaining the posture features of the smart device based on the authenticated user motion behavior data may be:

[0067] S21: Determine an error correction vector according to the linear acceleration data, the magnetic field data, and a preset quaternion.

[0068] Among them, the preset quaternion is used to represent the posture of the smart device. The preset quaternion is set to

[0069] The error correction vector is used to perform correction when acquiring the posture characteristics of the smart device. It mainly realizes the correction of the posture characteristics of the smart device by constraining the acceleration data and magnetic field data.

[0070] Specifically, the error correction vector is determined based on the collected linear acceleration data, magnetic field data, and a preset quaternion.

[0071] Optionally, based on the above embodiments, in some embodiments of the present invention, in order to reduce the dimensional impact of linear acceleration data and magnetic field data on obtaining the posture characteristics of the smart device, based on this.

[0072] Before executing S21, the following steps are also included:

[0073] S201: Initialize the preset quaternion.

[0074] Specifically, by assigning a value to the preset quaternion, the preset quaternion is initialized.

[0075] For example, Initialized to It indicates that the current initial posture of the smart device is parallel to the ground, but is not limited thereto. The present invention is not specifically limited thereto, and those skilled in the art may set it according to actual conditions.

[0076] S202: Normalize the linear acceleration data and the magnetic field data.

[0077] Specifically, after the linear acceleration data and the magnetic field data are obtained, normalization processing is performed on the linear acceleration data and the magnetic field data.

[0078] For example, for the linear acceleration data ax (t),a y (t),a z (t), and magnetic field strength data m x (t),m y (t),m z (t) The normalization process can be achieved according to the following normalization formula:

[0079]

[0080] Where acc=[a x (t),a y (t),a z (t)],mag=[m x (t),m y (t),m z (t)].

[0081] Optionally, based on the above embodiment, in some embodiments of the present invention, in order to reduce the impact of magnetic field data on the posture characteristics of the smart device, one implementation of S21 may be:

[0082] S211: Acquire first constraint data according to preset quaternion and magnetic field data.

[0083] The first constraint data is used to constrain the magnetic field data.

[0084] Specifically, first constraint data for constraining the magnetic field data is acquired according to a preset quaternion and magnetic field data.

[0085] Optionally, based on the above embodiments, in some embodiments of the present invention, after the magnetic field data is rotated from the body coordinate system to the geographic coordinate system using a preset formula, first constraint data corresponding to the magnetic field data in the horizontal and vertical directions are obtained.

[0086] Optionally, based on the above embodiment, in some embodiments of the present invention, the preset formula may be defined by the following expression:

[0087]

[0088] in, represents quaternion multiplication, Represents the conjugate of the preset quaternion, h x 、h y 、h z represents the three components of h, b x 、b z Indicates the first constraint data.

[0089] S212: Determine an error correction vector according to the linear acceleration data, the magnetic field data, the first constraint data, and the preset quaternion.

[0090] The error correction vector includes error correction components for the linear acceleration data and the magnetic field data in the x, y, and z directions. Based on this, the error correction vector is set to: in, represents the constraints on linear acceleration data, Represents a constraint on linear magnetic field data.

[0091] Optionally, based on the above embodiments, in some embodiments of the present invention, It can be qualified by the following expression:

[0092]

[0093] It can be qualified by the following expression:

[0094]

[0095] It should be noted that the error correction components of the linear acceleration data and the magnetic field data in the x, y, and z directions can respectively measure the difference between the theoretical value calculated by the current preset quaternion and the actual sensor data, thereby reducing the error of the linear acceleration data and the magnetic field data in obtaining the posture characteristics of the smart device.

[0096] S22: Obtaining a quaternion change rate of the preset quaternion within a preset time period according to the angular velocity data and the preset quaternion.

[0097] The quaternion change rate is used to characterize the instantaneous impact of angular velocity data on attitude characteristics. The quaternion change rate can be defined by the following expression:

[0098]

[0099] S23: Calculate the partial derivative of the error correction vector to obtain the error correction Jacobian matrix corresponding to the error correction vector.

[0100] Specifically, since the error correction vector is determined based on a preset quaternion, the partial derivative of the obtained error correction vector is obtained by taking the partial derivative of the preset quaternion, so as to obtain the error correction Jacobian matrix corresponding to the error correction vector.

[0101] S24: Update the preset quaternion according to the quaternion change rate, the error correction Jacobian matrix, and the error correction vector to obtain the target quaternion.

[0102] Optionally, based on the above embodiment, in some embodiments of the present invention, an implementation of S24 may be:

[0103] S241: Determine the quaternion update gradient according to the error correction Jacobian matrix and the error correction vector.

[0104] Among them, the quaternion update gradient can reduce the drift of posture features accumulated over time during the process of updating the preset quaternions at different acquisition times within a preset time length, thereby obtaining a more accurate target quaternion.

[0105] Specifically, the error correction Jacobian matrix is obtained by taking the partial derivative of the error correction vector, and the transpose of the error correction Jacobian matrix is multiplied by the error correction vector to obtain the quaternion update gradient.

[0106] Optionally, based on the above embodiment, in some embodiments of the present invention, the quaternion update gradient may be defined by the following expression:

[0107]

[0108] in, Represents the transpose of the error-corrected Jacobian matrix.

[0109] S242: Determine a quaternion update formula based on the quaternion update gradient and the quaternion change rate.

[0110] Optionally, based on the above embodiment, in some embodiments of the present invention, the quaternion update formula may be defined by the following expression:

[0111]

[0112] Among them, β represents the weight of the quaternion update gradient.

[0113] S243: Update the preset quaternion according to the quaternion update formula to obtain the target quaternion.

[0114] For example, the time difference between two adjacent acquisition moments of the motion sensor within a preset time period is determined to be Δt, then for the preset quaternion Update and obtain the target quaternion:

[0115] S25: Acquire posture features based on the target quaternion, linear acceleration data, angular velocity data, and magnetic field data.

[0116] Optionally, based on the above embodiment, in some embodiments of the present invention, an implementation of S25 may be:

[0117] S251: Obtain the Euler angle of the smart device's posture based on the target quaternion.

[0118] Specifically, after obtaining the target quaternion, the Euler angle of the attitude of the smart device is obtained according to the target quaternion.

[0119] The Euler angles of the posture include the rotation angle Yaw around the Z axis, the rotation angle Pitch around the Y axis, and the rotation angle Roll around the X axis. The rotation angles Yaw, Pitch, and Roll can be defined by the following expressions:

[0120] Roll angle:

[0121] Pitch angle: θ = arcsin(2(q0q2-q3q1))

[0122] Yaw:

[0123] S252: Obtaining the total acceleration and the total angular velocity respectively according to the linear acceleration data and the angular velocity data.

[0124] Optionally, based on the above embodiment, in some embodiments of the present invention, one implementation of S252 is to substitute the linear acceleration data and the angular velocity data into a preset formula to obtain the total acceleration and the total angular velocity. The preset formula may be defined by the following expression:

[0125]

[0126] S253: Obtain frequency domain attitude features and time domain attitude features according to the linear acceleration data, angular velocity data, magnetic field data, total acceleration, total angular velocity and attitude Euler angles.

[0127] Among them, the time domain posture feature refers to the posture feature of the smart device corresponding to the time domain, and the time domain posture feature includes: mean, extreme value (maximum and minimum), variance, absolute energy, and autocorrelation.

[0128] Frequency domain posture features refer to the posture features of smart devices corresponding to the frequency domain. The frequency domain posture features include: spectral centroid, spectral spread, spectral skewness, spectral kurtosis, power spectral density, and spectral entropy. The frequency domain posture features are determined by performing Fourier transform on linear acceleration data, angular velocity data, magnetic field data, total acceleration, total angular velocity, and posture Euler angle.

[0129] Optionally, based on the above embodiment, in some embodiments of the present invention, one implementation method for obtaining the user touch characteristics based on the authenticated user touch behavior data may be:

[0130] S31: Determine touch information corresponding to a plurality of user touch points of a preset duration.

[0131] The touch information includes: touch position coordinates, touch major axis, touch minor axis, and timestamp.

[0132] For example, the preset duration is the duration T of a sliding gesture performed by the user on the screen of the smart device, and the touch points of the sliding gesture within the preset duration T are P={P1, P2, ..., P n}, where each touch point P i ={(x i ,y i ),(l i ,w i ),t i}, but not limited to this, the present invention is not specifically limited, and those skilled in the art can set it according to actual conditions.

[0133] S32: Acquire user touch features at different preset stages within a preset time period according to the touch information.

[0134] The preset stages include: a touch sliding start stage, a touch sliding stable stage, and a touch sliding end stage. For example, following the above embodiment, for the preset duration T, and the touch points of the sliding gesture within the preset duration T are P = {P1, P2, ..., P n}, in the touch sliding startup phase, the touch point is determined to be [P1,P s ], in the touch sliding stable stage, the touch point is determined as [P s+1 ,P m ], at the end of the touch sliding phase, the touch point is determined to be [P m+1 ,P n ], but not limited to this, the present invention is not specifically limited, and those skilled in the art can set it according to actual conditions.

[0135] The user touch characteristics include at least: position characteristics, duration characteristics, length characteristics, direction characteristics, curvature characteristics, speed characteristics and touch area characteristics.

[0136] Specifically, the touch information corresponding to multiple user touch points of a preset duration is determined, such as touch position coordinates, touch major axis, minor axis, and timestamp. Based on the touch information, the user touch features of different preset stages within the preset duration are obtained, such as position features, duration features, length features, direction features, curvature features, speed features, and touch area features.

[0137] For example, following the above embodiment, the position feature may be, for example, the coordinates of the touch sliding starting point: P1 = (x1, y1) and the coordinates of the touch sliding ending point: P n =(x n,y n ).

[0138] For the duration characteristic Δt=t n -t1.

[0139] The length feature is used to describe the spatial coverage of the gesture, including displacement and cumulative distance. Cumulative distance i∈[1,n-1].

[0140] The direction feature is used to reflect the overall movement direction of the gesture. Figure 2 As shown, the 360-degree direction of the smart device screen is divided into 8 equally spaced intervals, and the direction angle θ between the starting point and the end point coordinates of the sliding is calculated as θ = atan2 (y n -y1,x n -x1),θ∈(-π,π], and assign a direction value according to the interval where the coordinate angle θ is located.

[0141] The curvature feature is used to represent the curvature of the sliding gesture trajectory.

[0142] The speed feature is used to represent the average speed of different preset stages during the sliding gesture, as well as the average speed within the entire preset duration, such as Figure 3 As shown, Figure 3 A schematic diagram of the speed of different touch points during three sliding processes of the same sliding gesture within a preset time T is provided in an embodiment of the present invention. Based on this, for the touch points of the sliding gesture within the preset time T, P = {P1, P2, ..., P n}, get the average speed of different preset stages, that is, the average speed in the touch sliding startup stage is The average speed during the touch sliding stable phase is The average speed at the end of the touch slide is The average speed during the entire preset time is

[0143] The touch area feature is used to represent the touch area at different preset stages during the sliding gesture and the touch area within the entire preset duration. Figure 4 As shown, Figure 4 A schematic diagram of the touch areas of different touch points during three sliding operations of the same sliding gesture within a preset time T is provided in an embodiment of the present invention. Based on this, the touch points of the sliding gesture within the preset time T are P = {P1, P2, ..., P n}, get the touch area of different preset stages, that is, the touch area in the touch sliding startup stage is The touch area in the touch sliding stable stage is The touch area at the end of the touch slide is The touch area during the entire preset duration is Among them, A i represents the area of the i-th touch point,

[0144] S12: Input the posture features and the user touch features into the identity authentication model to obtain the user's identity authentication result.

[0145] Among them, the identity authentication model is trained through the user's positive sample data, negative sample data and triple loss function. The positive sample data includes: the user's corresponding posture features, user touch features and labels, and the negative sample data includes: the non-user's corresponding posture features, user touch features and labels.

[0146] like Figure 5 As shown, the identity authentication model includes: a feature extraction module 10, a distance measurement module 20 and a decision classification module 30. The feature extraction module 10 is used to extract the identity features of the authenticated user, the distance measurement module 20 is used to calculate the similarity between the identity features and the positive sample data and the negative sample data, and the decision classification module 30 is used to obtain the identity authentication result based on the similarity.

[0147] Optionally, based on the above embodiment, in some embodiments of the present invention, before executing S12, the following steps are further included:

[0148] The user's positive sample data and negative sample data are input into the initial identity authentication model, and the model parameters are adjusted according to the triple loss function until the model converges to obtain a trained identity authentication model.

[0149] It should be noted that, continue to refer to Figure 5As shown, the feature extraction module 10 includes three identical one-dimensional convolutional network modules 1D-CNNs. The one-dimensional convolutional network module 1D-CNNs is a network model with a simple structure, and the three identical one-dimensional convolutional network modules 1D-CNNs share weight parameters, which can reduce the complexity and computational complexity of the network feature extraction process. Based on this, when the user's positive sample data and negative sample data are input into the initial identity authentication model for training, three inputs are included, namely Anchor Input, Positive Input and Negative Input. Among them, AnchorInput and Positive Input are both positive sample data of the user, that is, posture features and user touch features corresponding to the user, and Negative Input is negative sample data, that is, posture features and user touch features corresponding to non-users, that is, AnchorInput, Positive Input and Negative Input are respectively input into three identical one-dimensional convolutional network modules 1D-CNNs to realize model training.

[0150] The triple loss function can effectively cluster user identity features according to the input Anchor Input, Positive Input, and Negative Input during model training, thereby improving the performance of the model. Based on this, the triple loss function can be defined by the following expression:

[0151] POS dist =||ap|| 2

[0152] neg dist =||an|| 2

[0153]

[0154] Among them, a represents the identity feature vector corresponding to Anchor Input, p represents the identity feature vector corresponding to Positive Input, n represents the identity feature vector corresponding to Negative Input, and margin is used to dist When it is larger, force neg dist Bigger.

[0155] Optionally, based on the above embodiment, in some embodiments of the present invention, an implementation of S12 may be:

[0156] The posture features and the user touch features are input into a feature extraction module, and the identity features of the authenticated user are extracted by the feature extraction module.

[0157] The identity feature is input into the distance measurement module, and the first similarity between the identity feature and the positive sample data and the second similarity between the identity feature and the negative sample data are calculated by the distance measurement module.

[0158] The first similarity and the second similarity are input into a decision classification module, and the decision classification module performs classification to obtain an identity authentication result of the user.

[0159] Thus, the multimodal feature-based continuous identity authentication method provided in this embodiment uses the motion sensor on the smart device to collect user motion behavior data for a preset period of time, and uses the touch screen sensor to collect user touch behavior data for a preset period of time. Based on the authenticated user motion behavior data, the smart device's posture features are obtained, and based on the authenticated user touch behavior data, the user's touch features are obtained. The posture features and user touch features are input into the identity authentication model to obtain the user's identity authentication result. The identity authentication model is obtained by training the user's positive sample data, negative sample data and triple loss function. The positive sample data includes: posture features, user touch features and labels corresponding to the user, and the negative sample data includes: posture features, user touch features and labels corresponding to non-users. The triple loss function is used to achieve effective clustering of user identity features, thereby improving the accuracy of the identity authentication model in performing continuous identity authentication. Moreover, it can realize continuous identity authentication of users using smart devices based on the authenticated user's motion behavior data and the authenticated user's touch behavior data, solving the problem that the existing technology cannot be flexibly applied to other applications. The identity authentication model includes: a feature extraction module for extracting the identity features of the authenticated user, a distance measurement module for calculating the similarity between the identity features and the positive sample data and the negative sample data, and a decision classification module for obtaining the identity authentication result based on the similarity. In this way, when using the identity authentication model to realize continuous identity authentication, there is no need for multimodal feature fusion, avoiding the problem of excessive reliance on the acquisition, processing, modeling and learning of the user's behavioral biometric features in the existing technology, which has large model training calculation amount and high complexity.

[0160] It should be understood that although Figure 1-Figure 5 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. In addition, Figure 1-Figure 5 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0161] In one embodiment, Figure 6 As shown, a continuous identity authentication device based on multimodal features is provided, which is applied to a smart device. A motion sensor and a touch screen sensor are set on the smart device, including: a behavior data acquisition module 10, a feature acquisition module 11 and an identity authentication result acquisition module 12.

[0162] The behavior data acquisition module 10 is used to collect the authenticated user's motion behavior data within a preset time period through a motion sensor, and collect the authenticated user's touch behavior data within a preset time period through a touch screen sensor.

[0163] The feature acquisition module 11 is used to acquire the posture features of the smart device based on the authenticated user motion behavior data, and to acquire the user touch features based on the authenticated user touch behavior data.

[0164] The identity authentication result acquisition module 12 is used to input the posture features and the user touch features into the identity authentication model to obtain the user's identity authentication result.

[0165] The identity authentication model is trained using the user's positive sample data, negative sample data, and a triple loss function. The positive sample data includes: user-specific posture features, user touch features, and labels; the negative sample data includes: non-user-specific posture features, user touch features, and labels. The identity authentication model includes: a feature extraction module, a distance measurement module, and a decision classification module. The feature extraction module is used to extract the identity features of the authenticated user; the distance measurement module is used to calculate the similarity between the identity features and the positive sample data and negative sample data; and the decision classification module is used to obtain the identity authentication result based on the similarity.

[0166] In the above embodiment, the behavior data acquisition module collects the user's motion behavior data for a preset duration via a motion sensor, and collects the user's touch behavior data for a preset duration via a touchscreen sensor. The feature acquisition module obtains the smart device's posture features based on the user's motion behavior data, and obtains the user's touch features based on the user's touch behavior data. The identity authentication result acquisition module inputs the posture features and user touch features into the identity authentication model to obtain the user's identity authentication result. The identity authentication model is obtained by training the user's positive sample data, negative sample data and triple loss function. The positive sample data includes: posture features, user touch features and labels corresponding to the user, and the negative sample data includes: posture features, user touch features and labels corresponding to non-users. The triple loss function is used to achieve effective clustering of user identity features, thereby improving the accuracy of the identity authentication model in performing continuous identity authentication. Moreover, it can realize continuous identity authentication of users using smart devices based on the authenticated user's motion behavior data and the authenticated user's touch behavior data, solving the problem that the existing technology cannot be flexibly applied to other applications. The identity authentication model includes: a feature extraction module for extracting the identity features of the authenticated user, a distance measurement module for calculating the similarity between the identity features and the positive sample data and the negative sample data, and a decision classification module for obtaining the identity authentication result based on the similarity. In this way, when using the identity authentication model to realize continuous identity authentication, there is no need for multimodal feature fusion, avoiding the problem of excessive reliance on the acquisition, processing, modeling and learning of the user's behavioral biometric features in the existing technology, which has large model training calculation amount and high complexity.

[0167] For specific definitions of the multimodal feature-based continuous identity authentication device, please refer to the above-mentioned definitions of the multimodal feature-based continuous identity authentication method and will not be repeated here. Each module in the above-mentioned server can be implemented in whole or in part through software, hardware, or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory of the computer device in software form, so that the processor can call and execute the corresponding operations of each of the above modules.

[0168] An embodiment of the present invention provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the method for continuous identity authentication based on multimodal features provided by the embodiment of the present invention can be implemented. For example, when the processor executes the computer program, the method can be implemented. Figure 1-Figure 5 The technical solutions of any of the illustrated method embodiments have similar implementation principles and technical effects, which will not be described in detail here.

[0169] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, database or other media used in the embodiments provided by the present invention can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static random access memory (SRAM) and dynamic random access memory (DRAM).

[0170] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0171] The above-described embodiments merely illustrate several implementations of the present invention, and while their descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent. It should be noted that a person skilled in the art would be able to make numerous variations and improvements without departing from the spirit of the present invention, all of which fall within the scope of protection of the present invention. Therefore, the scope of protection of the patent for this invention shall be determined by the appended claims.

Claims

1. A continuous identity authentication method based on multimodal features, characterized in that: Applied to a smart device, wherein a motion sensor and a touch screen sensor are provided on the smart device, including: Collecting the user's motion behavior data for a preset time period through the motion sensor, and collecting the user's touch behavior data for a preset time period through the touch screen sensor; Acquire the posture characteristics of the smart device based on the authenticated user motion behavior data, and acquire the user touch characteristics based on the authenticated user touch behavior data; Inputting the gesture feature and the user touch feature into an identity authentication model to obtain an identity authentication result of the user; In which, the identity authentication model is obtained by training the user's positive sample data, negative sample data and triple loss function, the positive sample data includes: posture features, user touch features and labels corresponding to the user, and the negative sample data includes: posture features, user touch features and labels corresponding to non-users; the identity authentication model includes: a feature extraction module, a distance measurement module and a decision classification module, the feature extraction module is used to extract the identity features of the authenticated user, the distance measurement module is used to calculate the similarity between the identity features and the positive sample data and negative sample data, and the decision classification module is used to obtain the identity authentication result based on the similarity.

2. The method according to claim 1, characterized in that The authenticated user motion behavior data includes linear acceleration data, angular velocity data, and magnetic field data. The acquiring of the posture characteristics of the smart device based on the authenticated user motion behavior data includes: determining an error correction vector according to the linear acceleration data, the magnetic field data, and a preset quaternion; Obtaining a quaternion change rate of the preset quaternion within a preset time period according to the angular velocity data and the preset quaternion; Calculating a partial derivative of the error correction vector to obtain an error correction Jacobian matrix corresponding to the error correction vector; Update the preset quaternion according to the quaternion change rate, the error correction Jacobian matrix, and the error correction vector to obtain a target quaternion; The posture feature is acquired according to the target quaternion, the linear acceleration data, the angular velocity data, and the magnetic field data.

3. The method according to claim 2, characterized in that Before determining the error correction vector according to the linear acceleration data, the magnetic field data, and the preset quaternion, the method further includes: Initializing the preset quaternion; Normalization processing is performed on the linear acceleration data and the magnetic field data.

4. The method according to claim 3, characterized in that The determining of the error correction vector according to the linear acceleration data, the magnetic field data, and a preset quaternion includes: Acquire first constraint data according to the preset quaternion and the magnetic field data, wherein the first constraint data is used to constrain the magnetic field data; An error correction vector is determined according to the linear acceleration data, the magnetic field data, the first constraint data, and the preset quaternion.

5. The method according to claim 2, characterized in that The updating of the preset quaternion according to the quaternion change rate, the error correction Jacobian matrix, and the error correction vector to obtain a target quaternion includes: Determining a quaternion update gradient based on the error correction Jacobian matrix and the error correction vector; Determining a quaternion update formula according to the quaternion update gradient and the quaternion change rate; The preset quaternion is updated according to the quaternion update formula to obtain a target quaternion.

6. The method according to claim 5, characterized in that The acquiring the posture feature according to the target quaternion, the linear acceleration data, the angular velocity data, and the magnetic field data includes: Obtaining the Euler angle of the posture of the smart device according to the target quaternion; Obtaining a total acceleration and a total angular velocity respectively according to the linear acceleration data and the angular velocity data; Frequency domain attitude features and time domain attitude features are acquired according to the linear acceleration data, the angular velocity data, the magnetic field data, the total acceleration, the total angular velocity, and the attitude Euler angle.

7. The method according to claim 1, characterized in that The acquiring the user touch feature according to the authenticated user touch behavior data includes: Determine touch information corresponding to the plurality of user touch points of the preset duration, wherein the touch information includes: touch position coordinates, touch major axis, touch minor axis, and timestamp; According to the touch information, the user touch characteristics of different preset stages within the preset time length are obtained, wherein the preset stages include: a touch sliding start stage, a touch sliding stabilization stage, and a touch sliding end stage, and the user touch characteristics include at least: position characteristics, duration characteristics, length characteristics, direction characteristics, curvature characteristics, speed characteristics and touch area characteristics.

8. The method according to claim 1, characterized in that Inputting the posture feature and the user touch feature into an identity authentication model to obtain an identity authentication result of the user includes: Inputting the posture feature and the user touch feature into a feature extraction module, and extracting the identity feature of the authenticated user through the feature extraction module; Inputting the identity feature into the distance measurement module, and calculating a first similarity between the identity feature and the positive sample data, and a second similarity between the identity feature and the negative sample data by the distance measurement module; The first similarity and the second similarity are input into the decision classification module, and the decision classification module performs classification to obtain the user's identity authentication result.

9. A continuous identity authentication device based on multimodal features, characterized in that: Applied to a smart device, wherein a motion sensor and a touch screen sensor are provided on the smart device, including: A behavior data acquisition module, configured to collect the user's motion behavior data within a preset time period through the motion sensor, and collect the user's touch behavior data within a preset time period through the touch screen sensor; a feature acquisition module, configured to acquire the posture feature of the smart device based on the authenticated user motion behavior data, and acquire the user touch feature based on the authenticated user touch behavior data; An identity authentication result acquisition module, configured to input the posture feature and the user touch feature into an identity authentication model to obtain the user's identity authentication result; In which, the identity authentication model is obtained by training the user's positive sample data, negative sample data and triple loss function, the positive sample data includes: posture features, user touch features and labels corresponding to the user, and the negative sample data includes: posture features, user touch features and labels corresponding to non-users; the identity authentication model includes: a feature extraction module, a distance measurement module and a decision classification module, the feature extraction module is used to extract the identity features of the authenticated user, the distance measurement module is used to calculate the similarity between the identity features and the positive sample data and negative sample data, and the decision classification module is used to obtain the identity authentication result based on the similarity.

10. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

Images