Spatiotemporal deep learning for behavioral biometrics

Abstract

A behavioral biometric deep learning (BBDL) pipeline is provided that includes multiple stages operating to provide machine learning computer models of behavioral biometric authenticators operating on spatiotemporal input data. The BBDL pipeline receives spatiotemporal input data over multiple time intervals, each time interval having a corresponding subset of the spatiotemporal input data. For each time interval, a corresponding stage's machine learning computer model processes the subset of the spatiotemporal input data corresponding to that time interval to generate an output vector having values indicative of an internal representation of spatiotemporal characteristics of an entity. The output vectors are accumulated across the multiple stages of the BBDL pipeline to generate a final output vector indicative of spatiotemporal characteristics of an entity represented in the spatiotemporal input data. The entity is authenticated based on the final output vector.

Description

BACKGROUND

[0001] The present application relates generally to an improved data processing apparatus and method, and more specifically to mechanisms for spatiotemporal deep learning for behavioral biometrics.

[0002] Physical biometrics include the measurement and analysis of unique physical features (such as fingerprints, voice patterns, DNA, retinal patterns, etc.) that are typically used to verify the identity of a person. Behavioral biometrics is the field of study that deals with the measurement of unique identifying and measurable patterns in human activity. Examples of behavioral biometrics include keystroke dynamics (where patterns of rhythm and timing are created as a person selects keys on a keyboard device), gait analysis, computer mouse usage characteristics, signature analysis, etc. Behavioral biometrics are used for security authentication in many industries, including financial institutions, businesses, government facilities, retail point-of-sale (POS) devices, etc. SUMMARY

[0003] The summary is provided to present a simplified form of the concepts that are further described in the detailed description. The summary is not intended to identify key factors or essential features of the claimed subject matter, nor is it meant to be used to limit the scope of the claimed subject matter.

[0004] In one aspect of the present application, a method in a data processing system including at least one processor and at least one memory including instructions executed by the at least one processor to specifically configure the at least one processor to implement a behavioral biometric deep learning (BBDL) pipeline is provided, the BBDL pipeline including a plurality of stages of machine learning computer models operative to provide a behavioral biometric based authentication mechanism, the authentication mechanism operating based on spatiotemporal input data. According to one illustrative embodiment, the method includes receiving, from one or more sensors, spatiotemporal input data corresponding to input associated with an entity over a predetermined time window including a plurality of time intervals, each time interval having a corresponding subset of the spatiotemporal input data. The method further includes, for each time interval of the plurality of time intervals, processing the subset of the spatiotemporal input data corresponding to the time interval by one or more machine learning computer models of a respective stage of the plurality of stages to generate an output vector having values indicative of internal representations of spatiotemporal traits of the entity represented in the subset of the spatiotemporal input data. In addition, the method includes accumulating the output vectors over the plurality of stages of the BBDL pipeline to generate a final output vector including a final output vector value indicative of spatiotemporal traits of the entity represented in the spatiotemporal input data. Further, the method includes authenticating, by the data processing system, the entity based on the final output vector.

[0005] The spatio-temporal input data includes input data that is both spatially dependent on a physical area monitored by at least one of the one or more sensors and is time dependent. As such, the BBDL pipeline of illustrative embodiments does not require pre-processing of raw spatio-temporal input data to generate a fixed predetermined set of input features and remove temporal characteristics, such as in existing mechanisms. Moreover, processing of the spatio-temporal input data allows the BBDL pipeline to learn the behavioral biometrics of an entity (user) rather than merely authenticating an input pattern that can be provided by any entity or entities knowing the particular pattern. That is, the spatio-temporal input data represents unique behavior exhibited by the entity / user in the manner in which the entity / user generates the input and is not merely a detected pattern.

[0006] Preferably, the present invention provides a method wherein authenticating the entity includes comparing the final output vector to at least one previously generated output vector generated by the BBDL pipeline and stored in a user profile of an authorized user to determine a probability that the final output vector represents spatio-temporal characteristics of the authorized user. Moreover, authenticating the entity further includes controlling access to a protected resource in response to a result of the comparison. In this manner, a small set of previously generated output vectors can be used to define a user profile representing behavioral biometrics of an authorized user, and subsequent user input can be authenticated against such behavioral biometrics data.

[0007] Preferably, the present invention provides a method wherein controlling access to a protected resource includes: in response to determining that the probability indicates that the final output vector represents spatio-temporal characteristics of a user that match spatio-temporal characteristics of an authorized user: granting the user access to the protected resource; and updating the user profile to include the final output vector. In this manner, the user profile can be dynamically updated to reflect most current behavioral biometrics information of an authorized user.

[0008] Preferably, the present invention provides a method wherein each stage of the BBDL pipeline includes an image processing machine learning computer model, and wherein, for each stage of the BBDL pipeline, processing the subset of spatio-temporal input data corresponding to the time interval includes: processing the subset of spatio-temporal input data to transform spatio-temporal features in the subset of spatio-temporal input data into an image; and performing image analysis on the image by the image processing machine learning computer model of the stage to generate a first vector output. In this manner, spatio-temporal features can be represented in a format that can be subjected to image analysis by a machine learning model in order to classify spatio-temporal data with respect to various spatio-temporal features.

[0009] Preferably, the present application provides a method wherein each stage of the BBDL pipeline comprises a fully connected neural network machine learning computer model, and wherein, for each stage of the BBDL pipeline, processing the subset of spatiotemporal input data corresponding to the time interval comprises: processing the subset of spatiotemporal input data to identify temporal features in the subset of spatiotemporal input data; and processing the temporal features through the fully connected neural network machine learning model of the stage to generate a second vector output. Thereby, temporal features can be identified in the spatiotemporal input data, wherein the temporal features are independent of particular spatial elements, and can be evaluated separately from spatiotemporal features prior to combination of results of analyzing the spatiotemporal features.

[0010] Preferably, the present application provides a method wherein each stage of the BBDL pipeline comprises fixed / classification data embedding logic, and wherein, for each stage of the BBDL pipeline, processing the subset of spatiotemporal input data corresponding to the time interval comprises: processing the subset of spatiotemporal input data to identify fixed / classification features in the subset of spatiotemporal input data; and processing the fixed / classification features by the fixed / classification data embedding logic of the stage to generate a third vector output. Thus, the mechanism of illustrative embodiments is able to further base evaluation of input data on fixed / classification data as well as spatiotemporal features and temporal features in the spatiotemporal input data.

[0011] Preferably, the present application provides a method wherein each stage of the BBDL pipeline comprises combination logic operative to combine the first output vector, the second output vector, and the third output vector to generate a combined output vector that is input to the primary stage machine learning computer model of the stage. Thereby, results of processing spatiotemporal features, temporal features, and fixed / classification features in the spatiotemporal input data are combined to generate an internal representation of input spatiotemporal features for use in predicting whether the spatiotemporal input data is associated with an authorized user.

[0012] Preferably, the present application provides a method wherein each primary stage machine learning computer model of each stage of the BBDL pipeline processes the corresponding combined output vector of its corresponding stage along with an input from a preceding stage of the BBDL pipeline to generate a stage output vector, and wherein accumulating output vectors across the plurality of stages of the BBDL pipeline to generate a final output vector comprises, for each stage, outputting the stage output vector as an input to a next stage in the BBDL pipeline, and wherein the stage output vector of the primary stage machine learning computer model of a last stage of the BBDL pipeline is the final output vector. In this way, processing of spatiotemporal features, temporal features, and fixed / classification features across multiple time intervals of a time window is accumulated, enabling identification of behavioral biometric patterns within the time window.

[0013] Preferably, the present application provides a method wherein the embedded device information is input to a primary stage machine learning computer model of a first stage of a BBDL pipeline, and wherein the primary stage machine learning computer model processes the combined output vector associated with the first stage along with the embedded device information to generate a stage output vector for the first stage. Thus, the evaluation of the BBDL pipeline can be tailored to a particular device associated with the spatiotemporal input data.

[0014] Preferably, the present application provides a method wherein the one or more sensors comprise touch sensors of a touch-sensitive display device, and wherein the spatiotemporal input data comprises sensor data indicative of characteristics of a touch input by the entity. For example, the mechanisms of illustrative embodiments can be implemented in conjunction with touch-sensitive display devices on many modern electronic devices, such as smartphones, tablet computers, and the like.

[0015] Preferably, the present application provides a computer program product comprising a computer-usable or readable medium having a computer readable program embodied therein. The computer readable program, when executed on a computing device, causes the computing device to perform various ones of and combinations of the operations outlined above with regard to the method illustrative embodiments.

[0016] According to another aspect of the present application, a system / apparatus is provided. The system / apparatus can include one or more processors and a memory coupled to the one or more processors. The memory can comprise instructions, which when executed by the one or more processors, cause the one or more processors to perform various ones of and combinations of the operations outlined above with regard to the method illustrative embodiments.

[0017] These and other features and advantages of the present application will be described in, or be apparent from, the following detailed description of the exemplary embodiments of the application. BRIEF DESCRIPTION OF DRAWINGS

[0018] The application will best be understood by reference to the following detailed description of illustrative embodiments thereof in conjunction with the accompanying drawings, of which:

[0019] Figures 1A-1D An example plot of spatiotemporal features that can be used for user authentication is shown in accordance with one illustrative embodiment;

[0020] Figure 1E An example plot of temporal features that can be used for user authentication is shown in accordance with one illustrative embodiment;

[0021] Figure 2 is an example block diagram of a behavioral biometric deep learning (BBDL) pipeline in accordance with one illustrative embodiment;

[0022] Figure 3 is an example block diagram illustrating a high-level representation of a training process for training a BBDL pipeline, according to one illustrative embodiment.

[0023] Figure 4 is an example block diagram illustrating a high-level representation of a runtime operation of a BBDL pipeline, according to one illustrative embodiment;

[0024] Figure 5 is a flowchart outlining example operations of a BBDL pipeline, according to one illustrative embodiment;

[0025] Figure 6 is an example diagram of a distributed data processing system in which aspects of the illustrative embodiments can be implemented; and

[0026] Figure 7 is an example block diagram of a computing device that can implement aspects of the illustrative embodiments. DETAILED DESCRIPTION

[0027] Behavioral biometrics is an authentication paradigm that allows continuous authentication, as opposed to one-time authentication typically employed by physical biometrics and knowledge-based (e.g., password-based, security question-based) authentication mechanisms. For example, on a smart phone, tablet device, etc., different users can have different swiping speeds, pinching angles, etc., which can be monitored by a behavioral biometric mechanism. While there can be mechanisms that use behavioral biometrics as an authentication method, these existing mechanisms only consider pre-defined features with fixed dimensions that have eliminated the time dimension due to the limitations of these existing methods that are not able to handle variable length input features.

[0028] That is, existing mechanisms require fixed dimensions in order to train a computer model to perform prediction or classification, however, the time aspect of behavioral biometrics is variable. As such, the time aspect of the raw input data must be reduced from the input data processed by the computer model either by pre-processing or pre-computing the raw input data in order to generate defined higher level features that specifically distinguish one entity from another, for example, a swiping angle feature can be defined that involves computing the angle of a swipe and representing it as a Boolean value that depends on a threshold, then this can be used to determine whether the angle of the swipe represents an angular turn in the swipe or a circular turn, e.g., 1 if it is an angular swipe or 0 if it is not; 1 if it is a circular turn or 0 if it is not. These features can be able to distinguish users by the type of turn used by the user in performing the swiping gesture, however, the features themselves do not contain any time aspect or specific spatial information.

[0029] It is this pre-processed data, including advanced entity distinguishing features, that is processed by existing behavioral biometric computer models, which are generated by transforming the raw input data into a fixed dimension input with pre-defined entity distinguishing features, where the variable timing and / or spatial aspects of the raw input data are removed. Due to the limitations of existing behavioral biometric computer models requiring fixed dimension input, such advanced features with fixed dimension are required. As a result, existing mechanisms do not perform their behavioral biometric classification operations based on the raw spatio-temporal input data, as they are unable to account for the variability of the temporal dimension with respect to behavioral biometrics.

[0030] As an example, raw input data from a sensor device, such as a stroke or swipe by a user on a touch sensitive screen, can be a function of time and space F(t, x, y) = v, which indicates a value of a characteristic of the stroke / swipe sensed at coordinate (x, y) at time t (or alternatively F(t) = (x, y, v), where x is a horizontal coordinate of the screen, y is a vertical coordinate of the screen, t is a point in time, and v is a value of a sensed characteristic of the stroke or swipe (e.g., pressure, etc.). Existing mechanisms are unable to handle such variable input, as computer models based on fixed dimension must be trained using fixed dimension feature input data. Instead, the input data must be converted into fixed dimension user distinguishing features, e.g., Boolean values, through feature extraction transformations implemented by pre-processing of the input data, such as converting F(t, x, y) = v into Boolean input values by pre-processing the input such that F(x, y) = 1 or 0 depending on whether the screen is touched at coordinate (x, y), thereby removing the temporal dimension of the input. That is, through feature extraction performed based on feature engineering, the feature extraction can generate F1(session) = \max | F(t, x, y) - avg(F(t, x, y)) | - min | F(t, x, y) - avg(F(t, x, y)) |, which effectively eliminates one or more dimensions t and / or x and y. max(F - avg(F)) tells how big a certain feature can be during a session, such as how fast a swipe can be compared to the session average, where a large number means a large deviation. This equation can be further elucidated using the 2-norm instead of absolute values, e.g., F1(session) = \max || F(t, x, y) avg(F(t, x, y)) ||_2 - min || F(t, x, y) - avg(F(t, x, y)) ||_2, where “||_2” denotes the 2-norm.

[0031] Such preprocessed inputs generate fixed-dimensional values ​​representing high-level features of the input, such as "this user forms acute angles", "this user zooms in quickly / slowly", "this user zooms in and out alternately / repeatedly", etc. These fixed-dimensional values ​​are then fed into a computer model that performs classification or prediction based on the high-level features from which spatiotemporal input data has been removed.

[0032] The illustrative embodiment provides a behavioral biometrics deep learning (BBDL) computer model pipeline (hereinafter referred to as the BBDL pipeline) including a machine learning image analysis computer model, a feature classification machine learning model, a feature embedding mechanism, etc., which operate to learn patterns of the spatiotemporal characteristics of raw spatiotemporal input data without removing temporal and / or spatial characteristics to generate high-level features as in existing mechanisms. These learned patterns of spatiotemporal characteristics instruct behavioral biometrics for a specific user and generate a user profile based on the spatiotemporal feature patterns from the machine learning. This user profile can then be used to authenticate subsequent user inputs based on the spatiotemporal feature representations of subsequent user inputs. It should be understood that the raw spatiotemporal input of the BBDL pipeline is raw input data that is not designed itself to have specific entity-distinguishing features; that is, the raw input data is not preprocessed input data for feature engineering-based feature extraction and still includes raw temporal and spatial information. That is, unlike existing mechanisms where computer models must operate on inputs designed specifically to distinguish particular entities, the inputs to the BBDL pipeline are any raw input data that happens to be used in a particular embodiment, and include raw, unprocessed spatiotemporal input data from one or more sensor devices over a particular sliding time window.

[0033] The time window in which raw input data is collected and fed into the BBDL pipeline is a sliding or moving time window, where each individual time interval may or may not overlap. For example, if the time window includes time points T1, ..., T12, then individual time intervals (or time segments) can be (T1, T2), (T2, T3), ... (T11, T12) or (T1, T3), (T2, T4), (T3, T5), ... (T10, T12), depending on whether overlapping time intervals are expected for a particular implementation. In this example, observable (detectable by one or more sensors) input events from an entity (assumed to be human users for illustrative purposes, but not limited to) can occur at any time between T1 and T12. That is, if T1 = 12:00:10 (hours:minutes:seconds) and T2 = 12:00:15 (based on wall clock time), then the swipe event can occur at 12:00:12, which can be associated with the first time interval (T1, T2). By overlapping the time intervals, the possibility of information loss is minimized. For example, without overlap, two events at 12:00:11 and 12:00:13 would fall into the same time interval; however, despite the same time difference, two events at 12:00:14 and 12:00:16 would fall into two different time intervals.

[0034] Therefore, the BBDL pipeline operates on raw spatiotemporal input data that is not directly related to specific high-level features of entity behavior. Thus, the BBDL pipeline mechanism can be dynamically trained for new classifications (e.g., new users) with only a very small set of examples, such as embedding two swipe sessions and comparing the similarity of the embeddings, as described in more detail later. In contrast, existing mechanisms require training a computer model for a fixed set of classifications, or separately trained computer models for different classifications, and if a new classification is desired, the entire computer model must be retrained with the new classification or a new computer model must be generated.

[0035] According to one illustrative embodiment, the BBDL pipeline includes multiple stages of computer logic, each stage including logic that implements a set of machine learning models (or "sub-models" when the BBDL pipeline as a whole is considered as a computer "model"), which evaluate spatiotemporal features, temporal features, and fixed / classification features for specific corresponding time intervals including time points t, t+1, t+2, to t+n, where n indicates the size of the time window, on which spatiotemporal feature patterns are evaluated to determine behavioral biometrics for a particular user. Therefore, the BBDL pipeline will have n stages, each with multiple machine learning (ML) computer models (or “sub-models” of the overall BBDL pipeline model) that evaluate the spatiotemporal, temporal, and fixed / classification features of the user’s input to the computing device with which the user is connected. These features may be generated by one or more sensors of the user input device associated with the computing device, and / or by the computing device itself. For example, multiple sensors may be provided in association with one or more user-operated devices, such as smartphones, tablets, touchscreen devices, computer mice, trackball input devices, image capture devices, biometric reader devices, or any other source of biometric or behavioral biometric input data.

[0036] For example, in some illustrative embodiments, these sensors may include a touch sensor in a touch-sensitive screen where the input can be represented by F(t, x, y), an accelerometer associated with the device itself, an ambient light sensor, a camera device, etc. Examples of spatiotemporal characteristics sensed by such sensors and provided as data input to the BBDL pipeline include, but are not limited to, finger swipe data, such as swipe size, swipe speed, swipe acceleration, swipe direction, etc. on a touchscreen, including coordinate data relative to a particular touchscreen size, pressure information regarding touch pressure on a touchscreen device, mouse hover information, detected gesture information from an image acquisition device (camera, etc.), the touchscreen, or any other source of gesture information, voiceprint information associated with voice input, acceleration information representing the user's movement on a particular device, etc. In one illustrative embodiment, the spatiotemporal features used include Boolean touch (see description below), touch size (e.g., how large the area around a point (x, y) with a value of 1 for F(t, x, y) is), touch pressure, (x, y) touch velocity (determined from a sequence of points {(t1, x1, y1), (t2, x2, y2), ...} with velocities {(t1, (x2-x1) / (t2-t1), (y2-y1) / (t2-t1), ...}, and F(t, x, y) and F'(t, x, y) are constructed from these values), (x, y) touch acceleration, scalar touch velocity, and scalar touch acceleration. It should be understood that while Cartesian coordinates are used in the examples herein, the illustrative embodiment is not limited thereto, and any other coordinate system may be used without departing from the scope of the invention; for example, F(t, x, y) may be transformed into F(t, phi, r) to represent similar information (touch size, velocity, acceleration, etc.) in polar coordinates.

[0037] Generally, spatiotemporal features are a set of features that vary over time within the input range of a device (e.g., the screen size of a touchscreen device or the size of a graphical user interface, or the movement of a computer mouse, device trackball, or other user input device representing user input for a graphical user interface). For example, Boolean spatiotemporal touch features are a set of features representing which part of the device screen is touched at a specific time step. Formally, this can be represented as $f_{Boolean touch}(x, y, t)$, where the value is 1 if the device screen is touched at (x, y) at time step t, and 0 otherwise. Thus, for each value of (x, y) at time step t, the spatiotemporal feature has a value of 1 or 0, resulting in a matrix or bitmap representation of the user input. In the case of the function F(t, x, y) = v, the value of v is not limited to Boolean values, but depends on the sensed input being within a predefined range of values ​​for v, for example, v can be between max(v) and min(v) (e.g., 1 and -1).

[0038] The use of spatiotemporal features according to the illustrative embodiments differs significantly from previous gesture-based authentication mechanisms, which ignore position relative to the input range of the device and the fact that gestures change over time as the user performs actions (previous gesture-based authentication mechanisms only involved eliminating high-level features such as spatial and / or temporal characteristics as previously discussed). Previous gesture-based authentication mechanisms memorized a fixed set of gestures and authenticated based on the input of the same or different sets of gestures, such that if the input had exactly the same set of gestures, the user was authenticated, and if not, the user was not authenticated. Therefore, anyone who knew the memorized fixed pattern could be successfully authenticated. On the other hand, the illustrative embodiments use raw input data and operate on subtle differences in user (entity) behavior that are not associated with a fixed pattern. That is, subtle differences in user behavior provide a representation of the spatiotemporal characteristics of a particular user (e.g., spatiotemporal user input characteristics such as the speed of drawing curves, scaling angles, and trends) used with various inputs. These characteristics are captured with respect to time information and are uniquely identifiable to a particular user, rather than fixed patterns of non-spatiotemporal features that can be copied by any user who knows the fixed pattern of input.

[0039] The illustrative embodiments consider temporal characteristics that are independent of the input range associated with a particular device, which can be aggregated spatiotemporal information or non-spatiotemporal information associated with user input. For example, accelerometer changes are independent of a specific input range of the device, such as the size of the touchscreen of the device to which the user provides input, but simply relative to previous measurements of acceleration. That is, the change in acceleration will be the same regardless of the input range of a particular device. Formally, $f_{accelerometer-x}(t)$ is the change of the accelerometer along the x-axis at a time step t. Such temporal characteristics include, but are not limited to, three-dimensional accelerometer values ​​in Euclidean / polar coordinates, first derivatives of acceleration, second derivatives of acceleration, aggregated values ​​of spatiotemporal characteristics, and contextual embeddings, such as which components of the graphical user interface and / or touchscreen are touched.

[0040] In addition to spatiotemporal and temporal input data, the illustrative embodiments also obtain data from information sources that are independent of biometric data, behavioral biometric data, or other data associated with user input. Examples include other characteristics independent of such user input-specific data, such as the identity of the device itself being manipulated by the user, for example, brand and model (e.g., Apple iPhone X, Samsung Galaxy S20, etc.), device configuration (e.g., sensor type and their characteristics), device size or maneuverability limitations (e.g., screen size, touch panel size, graphical user interface size, etc.), the specific application used by the entity / user to interface for input, etc. These characteristics are non-changing over time; that is, they are fixed characteristics, but are useful in combination with spatiotemporal and temporal characteristics used for user authentication. Other examples of usable fixed characteristics include configuration information indicating the range of measurable values; for example, different devices may use different value ranges, different units, etc., to measure input.

[0041] These features can be embedded as one or more vector representations of the device itself. Any feature of the classification can be embedded, and the embedded vectors are provided as a whole as input to the rest of the network for that level and / or the BBDL pipeline. Examples of other features that can be embedded in this way include, but are not limited to, volume, speakerphone on / off, camera status, camera flash status, active / touch application, touch user interface elements, device name / type, operating system version, etc.

[0042] The source of feature data (e.g., spatiotemporal and temporal features) can be collected and / or generated continuously or periodically at multiple time points or time intervals, and the feature data can be provided to a preprocessor that operates to transform the raw feature data into a form usable by different machine learning models at various stages of the BBDL pipeline, such as a temporal x 2D feature matrix or mapping for spatiotemporal features, or a temporal x 1D matrix or mapping for temporal features. Furthermore, for any fixed device features or features of a classification type, the embedding of these features can be performed by the preprocessor or other embedding components, and the embedded vector representation can be provided along with the output of the neural network for downstream processing, as discussed below. Embedding is the process of generating a vector representation of an input value. In neural network mechanisms, embedding is the process of generating a low-dimensional, learned continuous vector representation of a discrete variable. The process of embedding inputs into vector representations is generally known in the art.

[0043] In some illustrative embodiments, each stage of the BBDL pipeline includes one or more first ML computer models (or “sub-models”) trained via machine learning processes to perform image analysis or computer vision-based analysis on spatiotemporal feature data received from different sources and preprocessed by a preprocessor into a time x 2D feature matrix or mapping representation. The one or more first ML computer models may include a single ML computer model trained to perform classification or categorization of multiple different spatiotemporal features, or a combination of multiple ML computer models, each trained to perform classification or categorization of different spatiotemporal features. Where multiple ML computer models operate on the spatiotemporal feature data, the results of each ML computer model can be aggregated into a final vector output representation by merging logic of a machine learning model or another neural network that outputs the final vector output representation to the next stage of the BBDL pipeline, as described below.

[0044] It should be understood that the training of the ML computer models at each stage of the BBDL pipeline can be performed as part of a holistic training operation for the entire BBDL pipeline, and therefore, each individual model (or “sub-model”) at each stage need not be pre-trained, although this may be done in some exemplary embodiments. Thus, the BBDL pipeline (or “model”) can be trained by including the training of the computer models (or “sub-models”) at each stage as a holistic training operation for the entire BBDL pipeline. It should be understood that in some embodiments, one or more of the ML computer models or “sub-models” at each stage may be pre-trained or trained as part of a separate machine learning-based training operation (if desired), and training data for training the individual ML computer models or “sub-models” is available. For ease of description, the ML computer models at each stage of the BBDL pipeline will be referred to hereinafter as ML computer models rather than “sub-models,” and the BBDL pipeline will be referred to as the BBDL pipeline, not the BBDL pipeline ML computer model. However, it should be recognized that the BBDL pipeline is a holistic computer model with different levels, where each level comprises one or more “sub-models” of the holistic BBDL pipeline ML computer model.

[0045] Spatiotemporal feature data can be represented as a time x 2D feature matrix or mapping representation, essentially providing an image that can be processed by image analysis or computer vision neural networks or other image / visual analysis machine learning models to generate an output vector representing the classification of the input image. Examples of image / visual analysis machine learning models or "sub-models" in each stage of the overall BBDL pipeline computer model include AllConvNet, ResNet, Inception, and Xception. The image / visual analysis machine learning model of the illustrative embodiment is specifically trained via a machine learning process to classify or categorize the spatiotemporal matrix or mapping representation regarding specific behavioral biometric categories. Different classifications of different time x 2D feature matrices or mappings of different spatiotemporal features can be combined to generate a vector representation of the classification of spatiotemporal features present in the input at a specific time point or time interval.

[0046] In some illustrative embodiments of the invention, the final logit layer of the image / visual analytics machine learning model may be omitted. That is, in a neural network, layers produce features, which are then processed by dense layers to produce vectors with the same dimensions as the number of output categories. This is called a logit layer, and the final activation is applied on top of the output generated by the logit layer, for example, softmax activation. In some illustrative embodiments, these last two layers (i.e., the logit layer and the activation layer) are not used in the image / visual analytics of the illustrative embodiment, and therefore, the input to the logit layer is used as an internal representation of the image for the next stage in the BBDL pipeline.

[0047] Temporal features can be represented as a time x 1D matrix or mapping, which can be input to one or more second neural networks or trained machine learning computer models that generate a vector output representing the classification of the temporal features. In some illustrative embodiments, the one or more second neural networks or trained machine learning computer models can be one or more trained fully connected neural networks or other densely layered neural networks. The one or more second neural networks or trained machine learning computer models output one or more vector representations of the classification of the temporal feature input for the time interval, which can be combined with a spatiotemporal feature processing machine learning model and other embedded features for fixing or classifying (i.e., non-temporally dependent) features to generate a final vector output, which is input to a neural network or machine learning computer model (hereinafter referred to as a master-level machine learning (ML) model) for generating the output to the next stage of the BBDL pipeline.

[0048] It should be understood that the output of each level's model, and therefore each master-level ML model, is an internal representation learned as part of training the BBDL pipeline as a whole. That is, since the BBDL pipeline attempts to classify whether two input sessions come from the same user, each level's model will attempt to extract features that best represent the user's input image. Because these outputs of the level computer models are internal representations, they are not explicitly / individually trained to generate certain features, but rather trained as a whole as part of the overall machine learning-based training of the BBDL pipeline. The dimension of the output of each master-level ML model is the degree of spatiotemporal characteristics of the entity (user). For example, suppose dimension 1 represents the spatiotemporal characteristics of a user performing several consecutive rapid zooms in and out to examine the details of something on a smartphone screen (as opposed to a single zoom), while dimension 2 represents the characteristics of a single zoom. On both dimensions, 1 indicates that the performer of the input has that characteristic, and 0 indicates that they do not have that characteristic. The BBDL pipeline accumulates the characteristics of the master-level ML models detected over a time window. For example, the first master-level ML model can detect magnification, the second master-level ML model can detect shrinkage, the third master-level ML model can detect another magnification, and the fourth master-level ML model can detect shrinkage. Since these detections at each stage of the BBDL pipeline accumulate over a time window, the BBDL pipeline's determination dimension 1 should be 1 (not a single magnification feature).

[0049] In one illustrative embodiment, the master-level ML model of the first stage of the BBDL pipeline receives embeddings of device information and / or other fixed or categorical information not provided at each time step as input. For subsequent stages of the BBDL pipeline, the master-level ML model of the previous stage is coupled to the master-level ML model of the next stage and its output is provided as input to the master-level ML model of the next stage. In one illustrative embodiment, the master-level ML model of each stage of the BBDL pipeline can be a recurrent neural network (RNN), such as a Long Short-Term Memory (LSTM) RNN.

[0050] At each stage of the BBDL pipeline, the corresponding level ML model performs a similar function as described above, but for specific spatiotemporal, temporal, and other fixed / categorical features at a given time point or time interval. The output vector of the level ML model is combined with the input from the master level ML model of the previous level, serving as a feature input in the current level's ML model. This feature input is then processed by the master level ML model of that level to generate a vector output that is input to the master level ML model of the next level. This process continues to the final stage of the BBDL pipeline, which then generates an output vector K representing the user input over a time window with respect to spatiotemporal, temporal, and fixed / categorical inputs. The value of "K" indicates the number of values ​​included in the output vector representing the characteristics of the user input. This output vector K is unique for the behavioral biometric features of a specific user and can therefore be used to authenticate the user.

[0051] In one illustrative embodiment, any suitable machine learning procedure (e.g., supervised or unsupervised machine learning procedure) can be used to perform the training of machine learning models at each level of the BBDL pipeline. It should be understood that although the BBDL pipeline comprises multiple machine learning models at different levels, the BBDL pipeline machine learning training is performed as a whole with respect to the BBDL pipeline.

[0052] Generally, machine learning involves designing and developing techniques that take empirical data (such as network statistics and performance indicators) as input and identify complex patterns within that data. One pattern in machine learning techniques is to use an underlying model M, whose parameters are optimized to minimize a cost function associated with M given input data. For example, in the context of binary classification, model M could be a straight line that divides the data into two classes (e.g., labels) such that M = a*x + b*y + c, and the cost function would be the number of misclassified points. The learning process then operates by adjusting the parameters a, b, and c to minimize the number of misclassified points. After this optimization phase (or learning phase), model M can be used to classify new data points. Of course, this is a simplified example of binary classification, and other models that perform classification into more than two classes will use other similar techniques.

[0053] Using illustrative embodiments, the machine learning process generally involves learning how to represent a particular user's behavioral biometrics based on the spatiotemporal features, temporal features, and fixed / categorical features of user input. The machine learning process generally involves obtaining a pair of user session information representing user input within a specific time period or time window for each iteration of each epoch of machine learning training. This pair of user session information includes a first set of session information corresponding to the target user. The second session information in this pair may come from the same or different users. Label information corresponding to the second session information is obtained, indicating whether the user is the same or different. Both the first and second session information are processed separately through a BBDL pipeline to generate corresponding K-dimensional output vectors. The K-dimensional output vector of the first session information is compared with the K-dimensional output vector of the second session information to determine whether the session information is classified by the BBDL pipeline as belonging to the same or different users; that is, whether the session information of the two sessions is determined to be sufficiently similar to represent the same user.

[0054] The obtained determination can then be compared with the labels used for the second session information to determine whether the second session information actually comes from the same or different users. If the similarity result based on the labels of the second session information is incorrect, the operating parameters of the machine learning model are adjusted to reduce the error in the K value generated by the BBDL model. That is, if the similarity comparison indicates that the users of the two sessions are the same users, and they are different as indicated by the labels of the second session information, the operating parameters of the machine learning model are adjusted so that the generated K value is different for different users. If the similarity comparison indicates that the users of the two sessions are not the same, and they are the same as indicated by the labels of the second session information, the operating parameters of the machine learning model are adjusted so that the generated K value is more similar to different sessions with the same users. If the similarity comparison indicates that the users of the two sessions are the same, and they are the same as indicated by the labels of the second session information, no adjustment is needed. It should be recognized that this process is repeated for different second session information for the same and / or different users, for example, this process can be provided in the training data used to train the machine learning model for the BBDL pipeline.

[0055] Once trained, the BBDL pipeline can be executed on one or more user sessions of a target user to generate an authorized user profile that stores one or more K-valued outputs of the BBDL pipeline, each representing user input for an authorized user on a specific device. For example, historical session information for a user can be obtained for multiple user sessions, and the BBDL pipeline can process each of these session information to generate a K-dimensional output vector for each session stored in the user profile. This set of user sessions (i.e., the user's historical sessions) can be a continuously updated set of user historical sessions; for example, the last 10 sessions can be maintained, and the corresponding user profile can be dynamically updated as a new session with the user occurs. That is, for each subsequent user session, the BBDL pipeline executes on the input data from the user session to generate a corresponding K-dimensional output vector, which is then used to replace the oldest K-dimensional output vector entry in the user profile. Thus, a dynamic user profile is maintained for the authorized user.

[0056] Whether new user input for a new session actually comes from an authorized user can be determined in a manner similar to the previous description of training the BBDL pipeline, by determining whether the K-dimensional output vector of the new user input is sufficiently similar to the K-dimensional output vector information stored in the authorized user profile. "Sufficiently similar" can be determined relative to a predetermined threshold, indicating an acceptable amount of difference while still indicating that the two K-dimensional output vectors represent the same user (entity). In response to a comparison between the user profile and the K-dimensional output vector of the new user input leading to a determination that they are not sufficiently similar, the user is denied access to computing resources, physical locations, or any other resources protected by the behavioral biometric authentication mechanism of the illustrative embodiment. Furthermore, during runtime operation, in addition to granting access to protected resources, determining that the new user information for a new session comes from an authorized user can also cause the update engine to update the user profile using the K-dimensional output vector of the new session, such as by replacing the oldest K-dimensional output vector entry in the user profile if a predetermined number of K-dimensional output vector entries exist in the user profile.

[0057] Therefore, illustrative embodiments provide mechanisms for implementing spatiotemporal deep learning mechanisms for behavioral biometrics. An illustrative embodiment provides a multi-stage pipeline comprising a set of machine learning models for each stage, which classify or categorize input data at corresponding time points or time intervals with respect to spatiotemporal features, temporal features, and fixed / classification features. In some illustrative embodiments, spatiotemporal features are presented as image data, which is then processed by a trained image analysis or computer vision analysis machine learning model, while temporal features are processed by a machine learning model based on fully connected and / or dense layers. The stages of the pipeline operate on user input data at their corresponding time points or time intervals and on the output from the previous stage in the pipeline to ultimately generate a K-dimensional output vector representing the behavioral biometrics of the user input within the time window or time interval processed by the pipeline. This can be used to generate a user profile, which can then be used with subsequent user sessions to authenticate the user.

[0058] Before proceeding with the discussion of various aspects of the illustrative embodiments and the improved computer operations performed by the illustrative embodiments, it should be understood that throughout this specification, the term "mechanism" will be used to refer to elements of the invention that perform various operations, functions, etc. As used herein, the term "mechanism" can be an implementation of a function or aspect of an illustrative embodiment in the form of an apparatus, process, or computer program product. In the case of a process, the process is implemented by one or more devices, apparatuses, computers, data processing systems, etc. In the case of a computer program product, logic represented by computer code or instructions embodied in or on the computer program product is executed by one or more hardware devices to implement a function or perform an operation associated with a particular "mechanism". Therefore, the mechanisms described herein can be implemented as dedicated hardware, software executed on hardware (to thereby configure the hardware to perform the specific functions of the invention that the hardware would otherwise not be able to perform), software instructions stored on a medium (such that the instructions can be readily executed by the hardware to thereby specifically configure the hardware to perform the functions and particular computer operations described herein), a process or method for performing a function, or any combination of the above.

[0059] This specification and claims may use the terms "a," "at least one," and "one or more" to refer to specific features and elements of illustrative embodiments. It should be understood that these terms and phrases are intended to state that at least one particular feature or element is present in a particular illustrative embodiment, but more than one feature or element may also be present. That is, these terms / phrases are not intended to limit the specification or claims to the presence of a single feature / element or the requirement for the presence of multiple such features / elements. Rather, these terms / phrases require only at least a single feature / element, while multiple such features / elements may be within the scope of the specification and claims.

[0060] Furthermore, it should be understood that if the term "engine" is used herein with respect to the description of embodiments and features of the invention, such use is not intended to limit any particular implementation for implementing and / or performing actions, steps, processes, etc., attributed to and / or performed by that engine. An engine can be, but is not limited to, software, hardware, and / or firmware, or any combination thereof, that performs the specified function, including but not limited to any use of general-purpose and / or special-purpose processors combined with appropriate software loaded or stored in machine-readable memory and executed by a processor. Furthermore, unless otherwise specified, any name associated with a particular engine is for convenience of reference and is not intended to limit one to a particular implementation. Moreover, any function attributed to an engine can be performed by multiple engines, incorporated into and / or combined with the function of another engine of the same or different type, or distributed across one or more engines in various configurations.

[0061] Furthermore, it should be understood that the following description uses multiple different examples of various elements of the illustrative embodiments to further illustrate exemplary embodiments of the illustrative embodiments and to aid in understanding the mechanisms of the illustrative embodiments. These examples are intended to be non-limiting and are not exhaustive of the various possibilities for implementing the mechanisms of the illustrative embodiments. For those skilled in the art, in view of this specification, many other alternative implementations of these different elements exist besides or in lieu of the examples provided herein, without departing from the scope of the invention.

[0062] This invention can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions thereon for causing a processor to execute aspects of the invention.

[0063] Computer-readable storage media can be a tangible means for retaining and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example, but not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital universal disk (DVD), memory sticks, floppy disks, mechanical encoding devices such as punch cards or protrusions in slots having instructions recorded thereon, and any suitable combination of the foregoing. As used herein, computer-readable storage media should not be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses passing through fiber optic cables), or electrical signals transmitted through wires.

[0064] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a suitable computing / processing device via a network (e.g., the Internet, a local area network, a wide area network, and / or a wireless network), or to an external computer or external storage device. The network may include copper cables, optical fibers, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to a computer-readable storage medium within the suitable computing / processing device.

[0065] Computer-readable program instructions used to perform the operations of this invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​(such as Java, Smalltalk, C++, etc.) and conventional procedural programming languages ​​(such as the "C" programming language or similar programming languages). The computer-readable program instructions may be executed entirely on a user's computer, partially on a user's computer, as a standalone software package, partially on a user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer via any type of network (including a local area network (LAN) or a wide area network (WAN)) or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs) may be personalized to execute computer-readable program instructions by utilizing state information from the computer-readable program instructions in order to perform aspects of this invention.

[0066] The present invention will now be described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0067] These computer-readable program instructions may be provided to a processor of a computer or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / actions specified in one or more blocks of a flowchart and / or block diagram. These computer-readable program instructions may also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner, such that the computer-readable storage medium storing the instructions includes an article of manufacture containing instructions that implement aspects of the functions / actions specified in one or more blocks of a flowchart and / or block diagram.

[0068] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus, or other device to produce computer-implemented processing, such that the instructions executed on the computer, other programmable apparatus, or other device perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0069] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. Each block in a flowchart or block diagram may represent a module, segment, or portion of instructions, including one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than indicated in the figures. For example, depending on the functions involved, two consecutively shown blocks may actually be executed substantially simultaneously, or these blocks may sometimes be executed in reverse order. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action or executes a combination of dedicated hardware and computer instructions.

[0070] As described above, the mechanism of the illustrative embodiment provides a behavioral biometric deep learning (BBDL) pipeline that includes multiple levels of machine learning computer models that process input data with respect to spatiotemporal features, temporal features, and fixed / categorical features at different time steps or time intervals within a time window, thereby characterizing the input data into vector representations that can be used to generate user profiles and / or compare with user profiles to authenticate users. The BBDL pipeline includes a data preprocessor that acquires raw behavioral biometric data and transforms it into a format usable by the machine learning models at each level of the BBDL pipeline. For example, the preprocessor transforms spatiotemporal features present in the raw input data into a temporal x 2D feature matrix or mapping. The preprocessor also transforms the temporal features of the raw input data into a temporal x 1D feature matrix or mapping. Furthermore, the preprocessor may, either alone or in conjunction with a separate embedding engine, embed fixed / categorical input data into vector representations of embedded values ​​for use with the outputs of other machine learning models at corresponding logical levels of the BBDL pipeline.

[0071] Figures 1A-1D Example plots of spatiotemporal features that can be used for user authentication are shown according to an illustrative embodiment. In these examples, the plots of these spatiotemporal features are referred to as time x 2D feature plots and represent two-dimensional behavioral biometric features of user input over time. Figures 1A-1D Each time-x 2D feature plot includes a time dimension along the horizontal axis, where the two-dimensional (2D) characteristics of the spatiotemporal feature are plotted as points along the vertical axis, representing the first characteristic, and shade, or color, represents another characteristic of the 2D feature. The specific 2D feature plotted depends on the specific spatiotemporal feature being plotted. For example, from... Figures 1A-1D As can be seen, the plotting of spatiotemporal features yields an image representing the spatiotemporal features over time.

[0072] In the depicted time x 2D, Figures 1A-1D The feature map represents the spatiotemporal features associated with swiping user input on a touchscreen of a user-operated device (such as a smartphone, tablet, etc.). Figure 1A It is a time series of 2D feature maps representing the time of a swipe to input user data. The x and y axes correspond to the (x, y) coordinates on the touchscreen of the user-operated device and thus constitute a 2D plot. Each plot represents the user's input at a specific point in time or within a time interval. The chromaticity of the pixels in the 2D feature map represents the value of the feature at a specific coordinate, where the specific value and its meaning depend on the specific feature being represented. For example, a Boolean touch can be 1 or 0 at a specific coordinate (represented by two different colors), and velocity can be a real number represented by different chromaticities on the spectrum of possible chromaticities (e.g., from white to black), etc.

[0073] Figure 1B The time x 2D feature map in the image represents the 2D characteristics of the drawn swipe size relative to time. Figure 1B In drawing, 2D properties can be values ​​that indicate the range or area of ​​pixels around a given pixel touched by a user at a specific point in time or within a time interval.

[0074] Figure 1C The time x 2D feature map in the image represents the 2D characteristics of the drawn swipe speed relative to time. Figure 1C In the plot, 2D features can include swipe speed, which can be the x-component of speed, while another plot (not shown) can represent the y-component of speed. Chroma in the plot can represent positive / negative values, where the sign is relative to a specific axis. In the depicted plot, the chroma of a pixel indicates the left half of the motion from left to right (relative to the positive direction of the x-axis) and the right half from right to left (relative to the negative direction of the x-axis). Thus, the depicted motion is a pinch motion, such as when a user attempts to zoom out on a touchscreen. Similar to the above, Figure 1D The time x 2D feature map in the figure represents the 2D characteristics of the sweep acceleration relative to time.

[0075] Raw sensor data is provided as raw input to a preprocessor, which performs operations to transform spatiotemporal features into a time x 2D matrix or mapping, which is an image representing an individual spatiotemporal feature. The image data generated by the preprocessor as a result of generating the matrix or mapping of the time x 2D feature map can then be input to the corresponding level of the BBDL pipeline's image analytics / computer vision machine learning model. There can be a single image analytics / computer vision ML model for each spatiotemporal feature, or a single image analytics / computer vision ML model that processes image analytics / computer vision operations on multiple spatiotemporal feature matrices / matrices.

[0076] Additionally, as previously mentioned, the preprocessor can also convert the temporal features in the input data into a temporal x 1D feature matrix or mapping. These are features in the input data that do not vary within the device's input range (e.g., at the touchscreen's position), as described above, but change over time. Thus, the temporal x 1D feature matrix or mapping can be used to represent the rendering time and single-dimensional features of these temporal features, i.e., the values ​​corresponding to the temporal features. Figure 1E An example plot of temporal features that can be used for user authentication according to an illustrative embodiment is shown. Again, the horizontal axis represents time, while the vertical axis represents the values ​​of the temporal features, such as the x-axis recorded by an accelerometer. The dashed box represents a sliding time window used to sample the temporal features. A time x 1D matrix or mapping can be input into a fully connected or densely layered machine learning model that generates a vector output representing the temporal features. The output of the fully connected / densely layered machine learning model can be combined with the output of an image analysis / computer vision machine learning model and the embedding of fixed / classified input data to generate a combined vector output representation of the user input for that point in time or time interval, which can then be fed as input to a master-level ML model for processing.

[0077] Figure 2 This is an example block diagram of a behavioral biometric deep learning (BBDL) pipeline according to an illustrative embodiment. Figure 2 As shown and as described above, the BBDL pipeline 200 includes multiple stages 210-230, each stage 210-230 including a set of machine learning models 212-218, 222-228, and 232-238. These machine learning models evaluate spatiotemporal features, temporal features, and fixed / classifiable features for a specific corresponding time interval including one or more time points (e.g., t, t+1, t+2, up to t+n), where n indicates the size of the time window used to evaluate spatiotemporal feature patterns to determine the behavioral biometrics of a particular user. Therefore, the BBDL pipeline 200 will have n stages, each stage 210-230 having multiple trained machine learning (ML) computer models 212-218, 222-228, and 232-238 that evaluate the spatiotemporal, temporal, and fixed / classifiable features of user input received via an input device associated with the computing device to which the user is docked.

[0078] User input can be captured or collected by one or more sensors associated with a user input device associated with the computing device, and / or generated by the computing device itself. That is, according to one or more illustrative embodiments, a plurality of sensors 201-204 are provided in association with one or more user-operated devices (not shown), such as smartphones, tablets, touchscreen devices, computer mice, trackball input devices, image capture devices, biometric reader devices, or any other source of biometric or behavioral biometric input data. For example, in some illustrative embodiments, these sensors 201-204 may include touch sensors in a touch-sensitive screen, accelerometers associated with the device itself, etc. Examples of spatiotemporal features sensed by such sensors 201-204 and provided as part of input data 205-208 for input to the BBDL pipeline 200 include, but are not limited to, finger swipe data on a touchscreen (e.g., swipe size, swipe speed, swipe acceleration, swipe direction, etc.) including coordinate data relative to a specific touchscreen size, pressure information related to touch pressure on the touchscreen device, mouse hover information, detected gesture information from any other source of image acquisition device (camera, etc.), touchscreen, or gesture information, voiceprint information associated with voice input, acceleration information representing user movement on a specific device, etc. In one illustrative embodiment, the spatiotemporal features used include Boolean touch (see description below), touch size, (x, y) touch acceleration, scalar touch speed, and scalar touch acceleration.

[0079] Generally, spatiotemporal features are a set of features that vary over time across the input range of user interface devices 292-294 and / or computing device 290, such as the screen size of a device for a touchscreen device or computer mouse, the size of a graphical user interface, the movement of a trackball or other user input device representing user input for a graphical user interface, etc. Again, as mentioned above, example spatiotemporal features may be Boolean spatiotemporal touch features, which are a set of features representing which part of the device screen was touched at a certain time step, and may also be real-valued features having different values ​​along a potential value range, such as pressure values, etc. However, it should be understood that spatiotemporal features are not limited to features that can only be sensed by sensors of user input devices and / or computing devices. In some illustrative embodiments, other spatiotemporal features may also be evaluated, such as any spatiotemporal features corresponding to a physical area monitored by a sensor, for example, spatiotemporal data can be obtained from the physical area in the case of ambient light values, the sensor can detect light levels over a time window and in a specific physical area, and this light level data can represent spatiotemporal input data.

[0080] The illustrative embodiments also consider temporal characteristics that are independent of the input range associated with a particular user input device 292-295 or computing device 290. As described above, these temporal characteristics can be aggregated spatiotemporal information or non-spatiotemporal information associated with user input. Such temporal characteristics include, but are not limited to, three-dimensional accelerometer values ​​in Euclidean / polar coordinates, first derivatives of acceleration, second derivatives of acceleration, aggregated values ​​of spatiotemporal characteristics, and contextual embeddings, such as which components of the graphical user interface and / or touchscreen are touched.

[0081] Furthermore, for any fixed device feature or classification type feature, the embedding of these features can be performed by preprocessors 280, 282, 284 corresponding to specific levels 210-230 or other embedding components 216, 226, 236 of levels 210-230. Embedding components 216, 226, and 236 can be machine learning models trained to embed the corresponding fixed / classification features into vector representations representing these fixed / classification features.

[0082] It should be understood that sensors 201-204 provide raw input data 205-208, which includes spatiotemporal features, temporal features, and fixed / classification features for each of multiple time points or time intervals. These time points or time intervals together represent the time window of user input. The time window of user input can be a moving or sliding window, allowing sensors 201-204 to continuously monitor user input during a user session. Thus, at time point / interval t+n+1, time window processing restarts at stage 210, i.e., time point / interval t+n+1 becomes the time point / interval t of the new time window. As mentioned above, time intervals can be overlapping or non-overlapping.

[0083] Each raw input data 205-208 is provided to the corresponding preprocessors 280, 282, and 284 at specific logic levels 210-230. The preprocessors 280, 282, and 284 generate temporal x 2D feature matrices / maps, temporal x 1D matrices / maps, and fixed / classification features, which are provided to the corresponding machine learning models at logic levels 210-230, as previously described.

[0084] In addition to user input data, device data 250 can be obtained from computing device 290 and / or user interface device and embedded. For example, other device features that do not depend on such user input-specific data can be embedded as vector representations (embeddings) 260 and provided as input to the master-level ML model 218 of the first stage 210 of the BBDL pipeline 200. These device features include the identity of the device itself for the user to manipulate (e.g., brand and model (e.g., Apple iPhone X, Samsung Galaxy S20, etc.)), the device configuration (e.g., sensor type and its characteristics, etc.), the size or manipulable limitations of the device (e.g., screen size, touch panel size, graphical user interface size, etc.), and so on.

[0085] The sensors 201-204 of user input devices 292-294 and / or computing devices 290 continuously or periodically collect and / or generate raw input data 205-208, including spatiotemporal features, temporal features, and fixed / classifiable features, at multiple time points or time intervals. This feature data can be provided to corresponding preprocessors 280-284 associated with stages 210-230 of the BBDL pipeline 200. Preprocessors 280-282 operate to transform the raw user input data into forms available to different machine learning models 212-218, 222-228, and 232-238 of stages 210-230 of the BBDL pipeline 200, such as temporal x 2D feature matrices or mappings for spatiotemporal features, temporal x 1D matrices or mappings for temporal features, and fixed / classifiable features provided for embedding. Preprocessors 280-282 are configured to identify different types of features present in the raw input data 205-208. That is, preprocessors 280-284 can parse the raw input data 205-208 and, based on its configuration, classify the parts of the raw input data 205-208 as representing spatiotemporal features, temporal features, or fixed / categorical features. Based on the classification of the parts of the raw input data 205-208, the preprocessors 280-284 perform corresponding processing to generate a time x 2D matrix / mapping, a time x 1D matrix / mapping, and identifiers of fixed / categorical features that can be input for embedding. Then, the preprocessed user input data 240-244 is provided to the corresponding levels 210-230 logic for processing by the machine learning models of the corresponding levels 210-230.

[0086] like Figure 2As shown, each stage 210-230 of the BBDL pipeline 200 includes one or more first ML computer models 212, 222, 232, which are trained through a machine learning process to perform image analysis or computer vision-based analysis on spatiotemporal feature data present in the preprocessed user input data 240-244 for corresponding time points / segments. That is, one or more first ML computer models 212, 222, 232 may include a single ML computer model trained to perform classification or categorization of multiple different spatiotemporal features, or a combination of multiple ML computer models, each trained to perform classification or categorization on different spatiotemporal features. When multiple ML computer models operate on spatiotemporal feature data, the results of each ML computer model can be aggregated into a final vector output representation by merging logic (not shown). The merging logic concatenates or otherwise aggregates the outputs of the individual ML computer models 212. These outputs can then be combined by combinational logic 217, 227, 237 of stages 210-230 to generate a single final vector output representation of the master-level ML model 218 of the BBDL pipeline 200.

[0087] As discussed above, spatiotemporal feature data can be represented as a time x 2D feature matrix or mapping representation, essentially providing an image that can be processed by image analysis or computer vision neural networks or other image / visual analysis machine learning models to generate an output vector representing a classification of the input image. The image / visual analysis machine learning model of the illustrative embodiment is specifically trained through a machine learning process to classify or categorize spatiotemporal matrix or mapping representations of specific behavioral biometric categories. Different classifications of different time x 2D feature matrices / mappings of different spatiotemporal features can be combined or aggregated to generate a vector representation of the classification of spatiotemporal features present in the input at a specific time point or time interval.

[0088] The temporal features in the preprocessed user inputs 240-244 can be represented as a temporal x 1D matrix or mapping, which can be input to one or more second neural networks or trained machine learning computer models 214, 224, 234 that generate a vector output representing the classification of the temporal features. In some illustrative embodiments, the one or more second neural networks or trained machine learning computer models 214, 224, 234 can be one or more trained fully connected neural networks or other densely layered neural networks. One or more second neural networks or trained machine learning computer models 214, 224, 234 output one or more vector representations of the classification of temporal feature inputs for a time interval. These vector representations can be combined by combination logic 217, 227, 237 with the outputs from spatiotemporal feature processing machine learning models 212, 222, 232 and other embedded features generated by embedding logic 216, 226, 236 for fixed or categorical (i.e., non-temporally related) features to generate a final vector output. This final vector output is input to master-level machine learning (ML) models 218, 228, 238, which generate the output for the next stage of the BBDL pipeline 200.

[0089] The master-level ML model 218 of the first stage 210 of the BBDL pipeline 200 receives as input device information and / or other fixed or classification information 250 not provided at each time step. For subsequent stages 220-230 of the BBDL pipeline 200, the master-level ML model of the previous stage is coupled to the master-level ML model of the next stage and its output is provided as input to the master-level ML model of the next stage. For example, the output of master-level ML model 218 is input to master-level ML model 228 of stage 220 and the output of master-level ML model 228 is provided as input to master-level ML model 238. In an illustrative embodiment, the master-level ML models 218, 228, 238 of each of the stages 210-230 of the BBDL pipeline 200 may be recurrent neural networks (RNNs), such as Long Short-Term Memory (LSTM) RNNs.

[0090] At each stage 210-230 along the BBDL pipeline 200, the corresponding stage ML models 212-218, 222-228, and 232-238 perform similar functions as described above, but for specific spatiotemporal, temporal, and other fixed / classifiable features at corresponding time points or time intervals. The output vector of the stage ML model is combined with the input from the master stage ML model of the previous stage as a feature input in the current stage ML model, which is then processed by the master stage ML model of that stage to generate a vector output that is input to the master stage ML model of the next stage. This process continues until the final stage of the BBDL pipeline 200 (e.g., stage 230), which then generates an output vector K 270 representing the user input over a time window with respect to spatiotemporal, temporal, and fixed / classifiable inputs. In this way, the BBDL pipeline 200 accumulates the features of the entity (user) represented in the input at the time points represented in the time window. The resulting output vector K270 is unique for a specific user’s behavioral biometric features (or user features) and can therefore be used to authenticate users.

[0091] The training of machine learning models at each level of the BBDL pipeline 200 can be performed using any suitable machine learning process (e.g., supervised or unsupervised machine learning process). Figure 3 This is an example block diagram illustrating a high-level representation of the training process for training a BBDL pipeline 200 according to an illustrative embodiment. Using the illustrative embodiment, the machine learning process generally involves learning how to represent a particular user's behavioral biometrics according to the spatiotemporal features, temporal features, and fixed / categorical features of the user input. The machine learning process generally involves: for each iteration of each epoch of machine learning training, obtaining a pair of user session information 310 and 320 representing user input over a specific time period or time window. This pair of user session information 310 and 320 involves a first set of session information 310 corresponding to a target user, such as raw input data 205-208 obtained from different sensors and / or computing devices, and device information 250 of the user input device / computing device. The second session information 320 in this pair may come from the same or different users and may include other raw input data 305-308 and corresponding device information 250. A corresponding tag information 322 is obtained for the second session information 320, indicating whether the user is the same user or a different user. In some embodiments, each user session information 310 and 320 may have label data 312 and 322 specifying the identity of the user providing the user session information 310 and 320, such that for training purposes, a comparison of user identities can determine whether the users are the same or different.

[0092] After preprocessing by preprocessor 330 (which may include preprocessors 280-284), the first and second user session information 310 and 320 are processed by model 340, including BBDL pipeline 200, to generate corresponding K-dimensional output vectors 342 and 344, such as... Figure 2 The K-dimensional output vector 270 in the first session information is compared with the K-dimensional output vector 342 of the second session information to determine whether the session information is classified by the model 340 including the BBDL pipeline 200 as the same or different users, that is, whether the session information of the two sessions is determined to be sufficiently similar to represent the same user.

[0093] Comparison logic 350 compares the K-dimensional output vectors K1 and K2, 342, 344 to determine whether the output vectors K1 and K2 are sufficiently similar (the difference between the individual values ​​of the output vectors K1 and K2 is equal to or less than a predetermined or learned threshold) to conclude that the user providing user session information 310 is the same as or different from the user providing user session information 320. The comparison result generates a probability value indicating the likelihood that the user providing user session information 310 is the same as the user providing user session information 320, which can then be compared with a threshold probability. If the calculated probability meets or exceeds the threshold probability, it is predicted that the user providing user session information 310 is the same user providing user session information 320.

[0094] After obtaining the prediction, training logic 360 checks the accuracy of the prediction to determine whether the BBDL model 340, including the BBDL pipeline 200, generated the correct results for user session information 310, 320. For example, labels 312, 322 of user session information 310, 320 can be compared, or in an embodiment where only second user information 320 is labeled, label 322 can be compared with the identity of the authorized user to whom model 340 was trained. The comparison performed by training logic 360 determines whether the second session information actually comes from the same or different user. If the similarity prediction is incorrect based on labels 312 and / or 322 of user session information 310, 320, then training logic 360 adjusts the operating parameters of the machine learning model of the BBDL pipeline 200 of BBDL model 340 to reduce the error (loss or cost) in the K value generated by BBDL model 340.

[0095] It should be understood that the training of BBDL model 340 and BBDL pipeline 200 is user-independent because BBDL pipeline 200 and BBDL model 340 are trained to accurately generate K-vector outputs for instances of the same and different users. Training determines whether BBDL pipeline 200 and BBDL model 340 appropriately generate K-vector outputs that are sufficiently identical for the same user or sufficiently different for different users, so that predictions of user authentication are accurate based on behavioral biometric data of the user present in the user input. The mechanism of the illustrative embodiment provides a single BBDL pipeline 200 and BBDL model 340 that can operate on authorized user input to generate an authorized user profile, rather than requiring separate machine learning models to be trained for different users. Subsequently, BBDL pipeline 200 and BBDL model 340 can operate on new user input to generate a K-vector output, which can then be compared with a stored authorized user profile to authenticate that the user input comes from an authorized user.

[0096] That is, once trained, the BBDL model 340, including the BBDL pipeline 200, can be executed on one or more user sessions of the target user to generate an authorized user profile that stores one or more K-value outputs of the BBDL pipeline representing user input for an authorized user of a particular device. Figure 4 This is an example block diagram illustrating a high-level representation of the runtime operations of a BBDL pipeline according to an illustrative embodiment. Figure 4 As shown, user history session information 410 for multiple user sessions can be obtained, and each of these user history session information 410 can be processed by the BBDL pipeline 200 of the corresponding preprocessor 420 and model 430 to generate a K-dimensional output vector for each session stored in the user profile 440. This set of user sessions (i.e., user history sessions) can be a continuously updated collection of user history sessions 410; for example, the last 10 sessions can be maintained, and the corresponding user profile 440 can be dynamically updated when a new session with the user occurs. That is, for each subsequent user session, model 430, including the BBDL pipeline 200, performs an operation on the input data from the user session to generate a corresponding K-dimensional output vector, which is then used to replace the oldest K-dimensional output vector entry in the user profile 440. Therefore, a dynamic user profile 440 is maintained for authorized users.

[0097] Whether new user input information for a new session actually comes from an authorized user can be determined in a manner similar to that described previously regarding the training of model 340 and BBDL pipeline 200. An exception to the training operation is that, during the runtime operation after training, if it is determined that a user of a subsequent session does not match their claimed authorized user or the registered authorized user of the specific device from which they obtained session information, then authorization to access the protected resource can be denied. Specifically, the new user input information for the new session 450 processed by preprocessor 420 and model 430, including BBDL pipeline 200, produces a K-dimensional output vector 460, which can be compared with K-dimensional output vectors stored in user profile 440. K-dimensional output vector 460 can be compared with each K-dimensional output vector to determine whether K-dimensional output vector 460 sufficiently matches any of the K-dimensional output vectors stored in user profile 440. If K-dimensional output vector 460 sufficiently matches any of these stored K-dimensional output vectors in user profile 440, then the user providing the new session information 450 is determined to be an authorized user, and access to the protected resource can be authorized. If the K-dimensional output vector 460 does not sufficiently match any K-dimensional output vector stored in the user profile, then the user providing the new session information 450 is not an authorized user, and therefore, access to the protected resource can be denied. In addition to granting access to the protected resource, determining that the new session information 450 comes from an authorized user also allows the update engine 480 to use the K-dimensional output vector 460 of this new session information 450 to update the user profile 440, such as by replacing the oldest K-dimensional output vector entry in the user profile 440 if a predetermined number of K-dimensional output vector entries exist in the user profile 440, or by adding a new entry if the maximum number of entries has not yet been reached.

[0098] Therefore, as described above, illustrative embodiments provide mechanisms for implementing spatiotemporal deep learning mechanisms for behavioral biometrics. Illustrative embodiments provide a multi-stage pipeline comprising a set of machine learning models for each stage, which classify or categorize input data at corresponding time points or time intervals with respect to spatiotemporal features, temporal features, and fixed / classification features. In some illustrative embodiments, spatiotemporal features are presented as image data, which is then processed by a trained image analysis or computer vision analysis machine learning model, while temporal features are processed by a machine learning model based on fully connected and / or dense layers. The stages of the pipeline operate on user input data at corresponding time points or time intervals and the output from the previous stage in the pipeline to ultimately generate a K-dimensional output vector representing the behavioral biometrics of the user input within the time window or time period processed by the pipeline. This can be used to generate a user profile, which can then be used with subsequent user sessions to authenticate the user.

[0099] Figure 5 This is a flowchart outlining an example operation of a BBDL pipeline according to an illustrative embodiment. Figure 5 The operations outlined herein can be executed by the BBDL pipeline as part of runtime operations, assuming that the BBDL pipeline has been trained through the machine learning process described above.

[0100] like Figure 5 As shown, the operation begins by receiving user input for a new session (step 510). The user input is preprocessed to identify and preprocess spatiotemporal features, temporal features, and fixed / categorical features present in the received user input (step 520). This preprocessing may include the generation of temporal x 2D feature matrices / maps and temporal x 1D feature matrices / maps, as well as the identification of fixed / categorical features present in the received user input. For each time point of the time window represented in the user input, the user input is processed by the corresponding stage of the BBDL pipeline, and the results of each stage are combined to generate a K-vector output representation of the behavioral biometrics of the user input (step 530).

[0101] The identity of the authorized user of the device is determined, and the corresponding user profile of the authorized user is retrieved (step 540). The user's identity can be specified in the user input, such as a user identifier during the login process, which can be used to retrieve the user profile to find the identified user. In other cases, the device's identity can be used to perform a lookup operation in the authorized user's registry for a specific device.

[0102] A comparison is made between the K-vector entries in the user profile and the output representation of the K-vectors for the user input (step 550) to determine whether the vectors are sufficiently similar to indicate the same user, or insufficiently similar to indicate different users (step 560). If the prediction indicates the same user, access to the protected resource is granted (step 570), and the user profile is updated using the output of the K-vectors for the user input (step 580). If the prediction indicates different users, access to the protected resource is denied (step 590). The operation then terminates.

[0103] Illustrative embodiments can be used in many different types of data processing environments to protect various types of computing resources, physical locations, or devices based on behavioral biometrics, which are based on the analysis of spatiotemporal, temporal, and fixed / categorical characteristics of behavioral biometric inputs. Data processing environments may include single computing devices and / or distributed data processing environments. The following provides... Figure 6 and 7 This serves as an example environment in which various aspects of the illustrative embodiments can be implemented. It should be understood that... Figure 6 and 7This is merely an example and is not intended to assert or imply any limitation regarding the environment in which various aspects or embodiments of the invention may be practiced. Many modifications may be made to the depicted environment without departing from the scope of the invention.

[0104] Figure 6 A graphical representation of an example distributed data processing system that can implement aspects of the illustrative embodiments is depicted. The distributed data processing system 600 may include a network of computers, wherein aspects of the illustrative embodiments can be implemented. The distributed data processing system 600 includes at least one network 602, which is a medium for providing communication links between different devices and computers connected together within the distributed data processing system 600. The network 602 may include connections such as wired, wireless communication links, or fiber optic cables.

[0105] In the depicted example, servers 604 and 606, along with storage unit 608, are connected to network 602. Clients 610, 612, and 614 are also connected to network 602. These clients 610, 612, and 614 can be, for example, personal computers, network computers, etc. In the depicted example, server 604 provides clients 610, 612, and 614 with data such as boot files, operating system images, and applications. In the depicted example, clients 610, 612, and 614 are clients of server 604. The distributed data processing system 600 may include additional servers, clients, and other devices not shown.

[0106] In the illustrated example, the distributed data processing system 600 is an internet with network 602, which represents a global collection of networks and gateways communicating with each other using the Transmission Control Protocol / Internet Protocol (TCP / IP) protocol suite. The core of the internet is the backbone of high-speed data communication lines between master nodes or host computers, composed of thousands of routed data and messages from commercial, government, educational, and other computer systems. Of course, the distributed data processing system 600 can also be implemented to include many different types of networks, such as, for example, intranets, local area networks (LANs), wide area networks (WANs), etc. As described above, Figure 6 This is intended as an example, and not as an architectural limitation on different embodiments of the invention, and therefore, Figure 6 The specific elements shown should not be considered as limitations on the environment in which the illustrative embodiments of the invention can be implemented.

[0107] like Figure 6 As shown, one or more of the computing devices (e.g., server 604) can be specifically configured to implement behavioral biometric deep learning (BBDL) models including a BBDL pipeline, for example, including Figure 2BBDL production line 200 Figure 3 The BBDL model 340. The configuration of the computing device may include providing dedicated hardware, firmware, etc., to facilitate the execution of the operations described herein with respect to the illustrative embodiments and the generation of output. The configuration of the computing device may also, or alternatively, include providing a software application stored in one or more storage devices and loaded into the memory of the computing device (e.g., server 604), for causing one or more hardware processors of the computing device to execute the software application, the software application configuring the processors to perform operations and generate the output described herein with respect to the illustrative embodiments. Furthermore, any combination of dedicated hardware, firmware, software applications executed on hardware, etc., may be used without departing from the scope of the illustrative embodiments.

[0108] It should be understood that once a computing device is configured in one of these ways, it becomes a dedicated computing device specifically configured to implement the mechanisms of the illustrative embodiments and is not a general-purpose computing device. Furthermore, as described herein, the implementation of the mechanisms of the illustrative embodiments improves the functionality of the computing device and provides useful and concrete results that facilitate the evaluation of user input (such as user input 205-208) from user input devices (such as user input devices 205-208) of the client computing device 610. The user input represents behavioral biometrics regarding spatiotemporal features, temporal features, and fixed / categorical features. BBDL model 340 and BBDL pipeline 200 characterize the behavioral biometrics of the user input and generate a user profile, which can then be stored in user profile registry 630 for authorizing the user to access protected resources. The user profile in user profile registry 630 can be compared with the characterization of behavioral biometrics of subsequent user input to authenticate the user in subsequent sessions. Authentication logic 620 (such as...) Figure 4 The comparison and update logic (470 and 480) in the code compares, for example, behavioral biometric representations to authenticate the user, and if the user is not a genuine user, access to the protected resource can be denied. If the user is an authenticated user, then the user can be authorized to access the protected resource, and in some embodiments, the authenticated user's profile can be dynamically updated by authentication logic 620 (including, for example, update logic 470) with representations input by the session user.

[0109] In some embodiments, user input and other data characterizing the client computing device 610 itself can be received from sensors of user interface devices 292-295, which may be part of the client computing system 610. User input and / or device information can be provided via network 602 to a server 604 implementing a BBDL pipeline 200, which then processes the user input to authenticate the user. The server 604 can then return output authorizing or denying access to protected resources. It should be understood that, although in Figure 6 The client / server configuration is shown, but the illustrative embodiments are not limited thereto. Furthermore, the client computing devices 610-614 can take many different forms, such as client computers, tablet computers, smartphones, other smart devices, Internet of Things (IoT) devices, etc. In short, any computing device having a user input interface can be used without departing from the scope of the invention, through which behavioral biometric user input can be received and provided to the BBDL pipeline and BBDL model.

[0110] As described above, the mechanisms of the illustrative embodiments utilize specially configured computing devices or data processing systems to perform operations for implementing the BBDL pipeline 200 and performing user input authentication based on behavioral biometric deep learning. These computing devices or data processing systems may include various hardware elements that are specifically configured, through hardware configurations, software configurations, or combinations of hardware and software configurations, to implement one or more of the systems / subsystems described herein. Figure 7 This is a block diagram of only one example data processing system that can implement various aspects of the illustrative embodiments. Data processing system 700 is a computer (such as...) Figure 6 Examples of servers (604) include computer-usable code or instructions that can be located and / or executed to implement processes and aspects of illustrative embodiments of the invention in order to achieve the operation, output, and external effects of the illustrative embodiments described herein.

[0111] In the depicted example, the data processing system 700 employs a hub architecture including a northbridge and memory controller center (NB / MCH) 702 and a southbridge and input / output (I / O) controller center (SB / ICH) 704. A processing unit 706, main memory 708, and graphics processor 710 are connected to the NB / MCH 702. The graphics processor 710 can be connected to the NB / MCH 702 via an Accelerated Graphics Port (AGP).

[0112] In the depicted example, a local area network (LAN) adapter 712 is connected to SB / ICH 704. An audio adapter 716, a keyboard and mouse adapter 720, a modem 722, a read-only memory (ROM) 724, a hard disk drive (HDD) 726, a CD-ROM drive 730, a universal serial bus (USB) port and other communication ports 732, and a PCI / PCIe device 734 are connected to SB / ICH 704 via buses 738 and 740. The PCI / PCIe device may include, for example, an Ethernet adapter, an insert card, and a PC card for a notebook computer. PCI uses a card bus controller, while PCIe does not. ROM 724 may be, for example, a flash memory basic input / output system (BIOS).

[0113] HDD 726 and CD-ROM drive 730 are connected to SB / ICH 704 via bus 740. HDD 726 and CD-ROM drive 730 can use interfaces such as Integrated Drive Electronics (IDE) or Serial Advanced Technology Accessory (SATA). Super I / O (SIO) device 736 can be connected to SB / ICH 704.

[0114] The operating system runs on processing unit 706. The operating system coordinates and provides... Figure 7 The control of different components within the data processing system 700. As a client, the operating system can be a commercially available operating system, such as... Windows Object-oriented programming systems (such as Java) TM The programming system can run in conjunction with an operating system and execute from Java on the data processing system 700. TM A program or application provides a call to the operating system.

[0115] As a server, the data processing system 700 can, for example, run advanced interactive execution. Operating system or IBM eServer operating system TM System Computer systems, Power-based TM Computer systems with processors, etc. The data processing system 700 can be a symmetric multiprocessor (SMP) system that includes multiple processors in the processing unit 706. Alternatively, a single-processor system can be used.

[0116] The operating system, object-oriented programming system, and instructions for applications or programs reside on a storage device such as HDD 726 and can be loaded into main memory 708 for execution by processing unit 706. Processing in illustrative embodiments of the invention can be executed by processing unit 706 using computer-usable program code, which may reside in memory (e.g., main memory 708, ROM 724) or, for example, one or more peripheral devices 726 and 730.

[0117] Bus systems (such as) Figure 7 The bus 738 or bus 740 shown may consist of one or more buses. Of course, a bus system can be implemented using any type of communication structure or architecture that provides data transfer between different components or devices attached to a structure or architecture. Communication units (such as...) Figure 7 The modem 722 or network adapter 712 may include one or more devices for sending and receiving data. The memory may be, for example, main memory 708, ROM 724, or something similar. Figure 7 The cache is found in NB / MCH 702.

[0118] As described above, in some illustrative embodiments, the mechanism of the illustrative embodiments can be implemented as dedicated hardware, firmware, etc., and application software stored in a storage device (such as HDD 726) and loaded into a memory (such as main memory 708) for execution by one or more hardware processors (such as processing unit 706, etc.). Accordingly, Figure 7 The computing device shown is specifically configured to implement the mechanisms of the illustrative embodiments, and is specifically configured to perform the functions described herein. Figure 2 The BBDL pipeline 200 describes the operations and output generation, as well as the authentication mechanism based on the representation of user input generated by the BBDL pipeline 200.

[0119] Those skilled in the art will understand that Figure 6 and 7 The hardware within can vary depending on the implementation. (Except for or replacing...) Figure 6 and 7 The hardware described herein can be replaced with other internal hardware or peripheral devices, such as flash memory, equivalent non-volatile memory, or optical disc drives. Furthermore, the processes described in the exemplary embodiments can be applied to multiprocessor data processing systems other than the SMP systems mentioned above, without departing from the scope of the invention.

[0120] Furthermore, the data processing system 700 can take any form among many different data processing systems, including client computing devices, server computing devices, tablet computers, laptop computers, telephones or other communication devices, personal digital assistants (PDAs), etc. In some illustrative examples, the data processing system 700 can be a portable computing device configured with flash memory to provide non-volatile storage for storing operating system files and / or user-generated data. Essentially, the data processing system 700 can be any known or subsequently developed data processing system without architectural limitations.

[0121] As described above, it should be understood that illustrative embodiments may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment that includes both hardware and software elements. In one exemplary embodiment, the mechanisms of the illustrative embodiment are implemented in software or program code, which includes, but is not limited to, firmware, resident software, microcode, etc.

[0122] A data processing system suitable for storing and / or executing program code will include at least a processor, which is directly or indirectly coupled to memory elements via a communication bus, such as a system bus. Memory elements may include local memory used during the actual execution of the program code, mass storage, and cache memory providing temporary storage for at least some of the program code to reduce the number of times code must be retrieved from mass storage during execution. Memory can be of various types, including but not limited to ROM, PROM, EPROM, EEPROM, DRAM, SRAM, flash memory, solid-state memory, etc.

[0123] Input / output (I / O) devices (including but not limited to keyboards, displays, pointing devices, etc.) can be coupled to the system directly or via intermediate wired or wireless I / O interfaces and / or controllers. I / O devices can take many different forms besides conventional keyboards, displays, pointing devices, etc., such as communication devices coupled via wired or wireless connections, including but not limited to smartphones, tablet computers, touchscreen devices, voice recognition devices, etc. Any known or subsequently developed I / O devices are intended to be within the scope of the illustrative embodiments.

[0124] Network adapters can also be coupled to the system, enabling the data processing system to couple to other data processing systems or remote printers or storage devices via an intermediary private or public network. Modems, cable modems, and Ethernet cards are just a few of the currently available types of network adapters for wired communications. Wireless communication-based network adapters can also be used, including but not limited to 802.11a / b / g / n wireless communication adapters, Bluetooth wireless adapters, etc. Any known or subsequently developed network adapters are within the scope of this invention.

[0125] The invention has been presented for purposes of illustration and description and is not intended to be exhaustive or limited to the forms disclosed. Many modifications and variations will be apparent to those skilled in the art without departing from the scope of the described embodiments. The embodiments were chosen and described in order to best explain the principles of the invention, its practical application, and to enable others skilled in the art to understand various embodiments of the invention with various modifications suitable for the intended particular purpose. The terminology used herein has been chosen to best explain the principles of the embodiments, their practical application, or technical improvements to techniques found in the market, or to enable those skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method in a data processing system, the data processing system comprising at least one processor and at least one memory, the at least one memory including instructions executed by the at least one processor to specifically configure the at least one processor to implement a behavioral biometric deep learning (BBDL) pipeline, the BBDL pipeline including multiple stages of a machine learning computer model operating to implement the method, the method comprising: Spatiotemporal input data corresponding to inputs associated with an entity are received from one or more sensors within a predetermined time window comprising multiple time intervals, each time interval having a corresponding subset of the spatiotemporal input data, wherein the spatiotemporal input data includes input data that is both spatially and temporally dependent with respect to a physical area monitored by at least one of the one or more sensors. For each of the plurality of time intervals, a subset of the spatiotemporal input data corresponding to the time interval is processed by one or more machine learning computer models of the corresponding level in the plurality of levels to generate an output vector, the output vector having a value indicating an internal representation of the spatiotemporal characteristics of the entity represented in the subset of the spatiotemporal input data; The output vectors are accumulated at the multiple stages of the BBDL pipeline to generate a final output vector, which includes a final output vector value indicating the spatiotemporal characteristics of the entity represented in the spatiotemporal input data; as well as The entity is authenticated by the data processing system based on the final output vector, wherein: Each stage of the BBDL pipeline includes combinational logic that operates to combine the output vectors of different logics within the stage and generate a combined output vector, which is then input to the master-level machine learning computer model of that stage. Each master-level machine learning computer model in each stage of the BBDL pipeline processes the corresponding combined output vector of its corresponding stage and the input from the previous stage of the BBDL pipeline to generate the stage output vector. Accumulating output vectors across the multiple stages of the BBDL pipeline to generate a final output vector includes, for each stage, using the output vector of the output stage as the input to the next stage in the BBDL pipeline, and The final output vector is the stage output vector of the master-level machine learning computer model in the last stage of the BBDL pipeline.

2. The method according to claim 1, wherein, Authenticating the entity includes: The final output vector is compared with at least one previously generated output vector from the BBDL pipeline stored in the authorized user's user profile to determine the probability that the final output vector represents the spatiotemporal characteristics of a user matching the spatiotemporal characteristics of the authorized user; and Access to the protected resource is controlled in response to the result of the comparison.

3. The method according to claim 2, wherein, Controlling access to protected resources includes, in response to determining the probability indicating that the final output vector represents the spatiotemporal characteristics of the user that match the spatiotemporal characteristics of the authorized user: Grant the user access to the protected resource; and Update the user profile to include the final output vector.

4. The method according to claim 1, wherein, Each stage of the BBDL pipeline includes an image processing machine learning computer model, and wherein, for each stage of the BBDL pipeline, processing a subset of spatiotemporal input data corresponding to the time interval includes: Processing a subset of the spatiotemporal input data to transform the spatiotemporal features of the subset of the spatiotemporal input data into an image; and The image is subjected to image analysis by the image processing machine learning computer model of the level to generate a first vector output.

5. The method according to claim 4, wherein, Each stage of the BBDL pipeline includes a fully connected neural network machine learning computer model, and wherein, for each stage of the BBDL pipeline, processing a subset of spatiotemporal input data corresponding to the time interval includes: Processing a subset of the spatiotemporal input data to identify temporal features within the subset of the spatiotemporal input data; and The time features are processed by the fully connected neural network machine learning model of the aforementioned level to generate a second vector output.

6. The method according to claim 5, wherein, Each stage of the BBDL pipeline includes fixed / categorical data embedding logic, and wherein, for each stage of the BBDL pipeline, processing a subset of spatiotemporal input data corresponding to the time interval includes: Processing a subset of the spatiotemporal input data to identify fixed / classification features within the subset of the spatiotemporal input data; and The fixed / classified features are processed by the fixed / classified data embedding logic of the level to generate a third vector output.

7. The method according to claim 6, wherein, For each stage of the BBDL pipeline, a corresponding combinational logic operation generates a combined output vector by combining a first output vector, a second output vector, and a third output vector, and the combined output vector is input to the master-level machine learning computer model of that stage.

8. The method according to claim 1, wherein, Embedded device information is input into the master-level machine learning computer model of the first stage of the BBDL pipeline, and wherein the master-level machine learning computer model processes the combined output vector associated with the first stage together with the embedded device information to generate the stage output vector of the first stage.

9. The method according to claim 1, wherein, The one or more sensors include touch sensors for touch-sensitive display devices, and wherein the spatiotemporal input data includes sensor data indicating the characteristics of touch input to the entity.

10. A computer program product comprising a computer-readable storage medium storing a computer-readable program, wherein the computer-readable program, when executed on a computing device, causes the computing device to implement a behavioral biometric deep learning (BBDL) pipeline, the BBDL pipeline comprising multiple stages of a machine learning computer model, the machine learning computer model operating to: Spatiotemporal input data corresponding to inputs associated with an entity are received from one or more sensors within a predetermined time window comprising multiple time intervals, each time interval having a corresponding subset of the spatiotemporal input data, wherein the spatiotemporal input data includes input data that is both spatially and temporally dependent with respect to a physical area monitored by at least one of the one or more sensors. For each of the plurality of time intervals, a subset of the spatiotemporal input data corresponding to the time interval is processed by one or more machine learning computer models of the corresponding level among the plurality of levels to generate an output vector, the output vector having values ​​indicating an internal representation of the spatiotemporal characteristics of the entity represented in the subset of the spatiotemporal input data; The output vectors are accumulated at the multiple stages of the BBDL pipeline to generate a final output vector, which includes a final output vector value indicating the spatiotemporal characteristics of the entity represented in the spatiotemporal input data; as well as The entity is authenticated based on the final output vector, wherein: Each stage of the BBDL pipeline includes combinational logic that operates to combine the output vectors of different logics within the stage and generate a combined output vector, which is then input to the master-level machine learning computer model of that stage. Each master-level machine learning computer model in each stage of the BBDL pipeline processes the corresponding combined output vector of its corresponding stage and the input from the previous stage of the BBDL pipeline to generate the stage output vector. Accumulating output vectors across the multiple stages of the BBDL pipeline to generate a final output vector includes, for each stage, using the output vector of the output stage as the input to the next stage in the BBDL pipeline, and The final output vector is the stage output vector of the master-level machine learning computer model in the last stage of the BBDL pipeline.

11. The computer program product according to claim 10, wherein, Authenticating the entity includes: The final output vector is compared with at least one previously generated output vector from the BBDL pipeline stored in the authorized user's user profile to determine the probability that the final output vector represents the spatiotemporal characteristics of a user matching the spatiotemporal characteristics of the authorized user; and Access to the protected resource is controlled in response to the result of the comparison.

12. The computer program product according to claim 11, wherein, Controlling access to protected resources includes, in response to determining the probability indicating that the final output vector represents the spatiotemporal characteristics of the user that match the spatiotemporal characteristics of the authorized user: Grant the user access to the protected resource; and Update the user profile to include the final output vector.

13. The computer program product according to claim 10, wherein, Each stage of the BBDL pipeline includes an image processing machine learning computer model, and wherein, for each stage of the BBDL pipeline, processing a subset of spatiotemporal input data corresponding to the time interval includes: Processing a subset of the spatiotemporal input data to transform the spatiotemporal features of the subset of the spatiotemporal input data into an image; and The image is subjected to image analysis by the image processing machine learning computer model of the level to generate a first vector output.

14. The computer program product according to claim 13, wherein, Each stage of the BBDL pipeline includes a fully connected neural network machine learning computer model, and wherein, for each stage of the BBDL pipeline, processing a subset of spatiotemporal input data corresponding to the time interval includes: Processing a subset of the spatiotemporal input data to identify temporal features within the subset of the spatiotemporal input data; and The time features are processed by the fully connected neural network machine learning model of the level to generate a second vector output.

15. The computer program product according to claim 14, wherein, Each stage of the BBDL pipeline includes fixed / categorical data embedding logic, and wherein, for each stage of the BBDL pipeline, processing a subset of spatiotemporal input data corresponding to the time interval includes: Processing a subset of the spatiotemporal input data to identify fixed / classification features within the subset of the spatiotemporal input data; and The fixed / classified features are processed by the fixed / classified data embedding logic of the level to generate a third vector output.

16. The computer program product according to claim 15, wherein, For each stage of the BBDL pipeline, a corresponding combinational logic operation generates a combined output vector by combining a first output vector, a second output vector, and a third output vector, and the combined output vector is input to the master-level machine learning computer model of that stage.

17. The computer program product according to claim 10, wherein, Embedded device information is input into the master-level machine learning computer model of the first stage of the BBDL pipeline, and wherein the master-level machine learning computer model processes the combined output vector associated with the first stage together with the embedded device information to generate the stage output vector of the first stage.

18. An apparatus comprising: At least one processor; as well as At least one memory coupled to the at least one processor, wherein the at least one memory includes instructions that, when executed by the at least one processor, cause the at least one processor to implement a behavioral biometric deep learning (BBDL) pipeline, the BBDL pipeline including multiple stages of a machine learning computer model, the machine learning computer model operating to: Spatiotemporal input data corresponding to inputs associated with an entity are received from one or more sensors within a predetermined time window comprising multiple time intervals, each time interval having a corresponding subset of the spatiotemporal input data, wherein the spatiotemporal input data includes input data that is both spatially and temporally dependent with respect to a physical area monitored by at least one of the one or more sensors. For each of the plurality of time intervals, a subset of the spatiotemporal input data corresponding to the time interval is processed by one or more machine learning computer models of the corresponding level among the plurality of levels to generate an output vector, the output vector having values ​​indicating an internal representation of the spatiotemporal characteristics of the entity represented in the subset of the spatiotemporal input data; The output vectors are accumulated at the multiple stages of the BBDL pipeline to generate a final output vector, which includes a final output vector value indicating the spatiotemporal characteristics of the entity represented in the spatiotemporal input data; as well as The entity is authenticated based on the final output vector, wherein: Each stage of the BBDL pipeline includes combinational logic that operates to combine the output vectors of different logics within the stage and generate a combined output vector, which is then input to the master-level machine learning computer model of that stage. Each master-level machine learning computer model in each stage of the BBDL pipeline processes the corresponding combined output vector of its corresponding stage and the input from the previous stage of the BBDL pipeline to generate the stage output vector. Accumulating output vectors across the multiple stages of the BBDL pipeline to generate a final output vector includes, for each stage, using the output vector of the output stage as the input to the next stage in the BBDL pipeline, and The final output vector is the stage output vector of the master-level machine learning computer model in the last stage of the BBDL pipeline.

Images