Dynamic Identity Authentication

Abstract

1. A method for identifying a person, the method comprising: acquiring spatiotemporal data for each of a plurality of anatomical landmarks associated with an activity involving the person, the anatomical landmarks determining a spatiotemporal trajectory of the anatomical landmarks during the activity; modeling the acquired spatiotemporal data as a space-time graph (ST-graph); and processing the ST-graph with at least one non-local graph convolutional neural network (NLGCN) to provide an identity corresponding to the person.

Description

Detailed Description of the Invention

[0001] Related Applications This application claims the benefit under 35 USC 119(e) of U.S. Provisional Application No. 63 / 067,890, filed August 20, 2020, the disclosure of which is incorporated herein by reference.

[0002] [Technical field] SUMMARY OF THE DISCLOSURE Embodiments of the present disclosure relate to methods and apparatus for providing biometric authentication of a person's identity.

[0003] [Background technology] An ever-growing list of services requires authentication procedures, conventionally called multi-factor authentication procedures (MFA), to authenticate and authorize user access to the services. MFA procedures require a user to provide an appropriate response to each of several categories of challenges. The challenge categories are called "authentication factors." Common MFA is called two-factor authentication (2FA), where the user is challenged to respond correctly to at least two of three authentication factors: a knowledge factor, a possession factor, and a unique factor. A knowledge factor tests something the user is expected to know, e.g., a password. A possession factor requires the presentation of something the user is expected to have, e.g., a credit card or smartphone. A unique factor requires the user to present something that characterizes the user, e.g., a biometric feature such as a fingerprint, voiceprint, or iris scan.

[0004] However, legacy authentication technologies appear to be facing difficulties in providing the ease of use and quality of authentication required by the increasing complexity and interdependence of the matrix of actions regularly performed by modern citizens. For example, legacy MFA configurations appear to be under severe pressure to meet the Strong Customer Authentication (SCA) specifications of the revised European Payment Services Directive (PSD2), which was promulgated to unite consumers, banks, and third-party providers (TPPs) in the Open Banking initiative. The implementation of SCA has been delayed twice. The regime, which was originally scheduled to start in September 2019, was postponed until March 14, 2021, and then to the current deadline of September 14, 2021.

[0005] Summary of the Invention One aspect of an embodiment of the present disclosure relates to providing a method, which may be referred to as a dynamic identification (DYNAMIDE) method, or simply DYNAMIDE, that identifies people based on the uniqueness of how they perform an activity. According to one embodiment of the present disclosure, DYNAMIDE includes identifying anatomical landmarks during an activity performed by people and identifying spatiotemporal trajectories of the anatomical landmarks while the activity is performed. The anatomical landmarks, optionally referred to as activity fiducials (AFIDs), indicate various degrees of movement or lack thereof. The spatiotemporal trajectories may be used to identify an activity. DYNAMIDE includes processing the trajectories to determine features of the trajectories that are advantageous for distinguishing activities performed by particular individuals and for identifying a particular individual performing the activity.

[0006] The characteristics of a behavior that may distinguish the behavior from the individual performing the behavior may be very subtle. The AFID trajectory associated with the behavior may exhibit substantially subtle and nonintuitive crosstalk. As a result, the characteristics of one spatiotemporal trajectory of a behavior that may appear intuitively independent of the characteristics of another spatiotemporal trajectory of the behavior may actually be unique to the individual performing the behavior and may provide a criterion for identifying the individual. According to an embodiment of the present disclosure, the provision of a spatiotemporal decision that is advantageous for discovering and using the uniqueness exhibited by the trajectory for trajectory identification and spatial and / or temporal processing may be non-local and multiple a priori processing constraints. The a priori processing constraints are assumed for the movement exhibited by the AFID trajectory that is advantageously limited.

[0007] According to one embodiment, identifying a specific individual based on a given behavior that the individual may perform includes acquiring a series of images of the individual performing the given behavior and identifying within the images an AFID associated with the given behavior. The images may be processed to determine a spatiotemporal trajectory exhibited by the identified AFID, the trajectory being processed to identify a specific individual who performed the behavior among a plurality of individuals who may perform the behavior. Optionally, processing the AFID trajectory includes determining local and non-local spatiotemporal correlations exhibited by the AFID during performance of the given behavior and using the correlations to determine the identity of the specific individual. The spatiotemporal correlations may include correlations based on spatial parameters, temporal parameters, or both spatial and temporal parameters that characterize a spatiotemporal trajectory or trajectory of one or more AFIDs.

[0008] In an embodiment of the present disclosure, the AFIDs associated with a given behavior can be anatomical landmarks of any body part, such as a limb, face, or head, that exhibit a spatiotemporal trajectory in the performance of the given behavior suitable for use in identifying the person performing the behavior. For example, the AFIDs can be the joints of the limbs or bones of the skeleton that exhibit a spatiotemporal trajectory that fits during behaviors such as walking, hitting a golf ball, or typing a password at an ATM. For typing, the AFIDS can include multiple joints to which the bones of the hand are connected. The AFIDs can be facial landmarks such as the corners of the eyebrows, eyes, and lips whose movements are used to define action units (AUs) of a facial action coding system (FACS). The facial action coding system is used to classify facial expressions and micro-expressions. The AFIDs can also be detailed paired features of fingerprints of multiple fingers of a hand, imaged non-contact with sufficient optical resolution to allow identification of paired features.

[0009] According to one embodiment, DYNAMIDE uses at least one neural network for processing images of the behavior to identify the individual performing the behavior. In one embodiment, the at least one neural network is trained to detect body parts or regions of interest (BROIs) in the images and identify the AFIDs they may contain. The spatial and temporal progression of the identified AFIDs during the performance of the behavior is represented by a spatiotemporal graph (ST-Graph). In the spatiotemporal graph, the AFIDs are nodes connected by spatial and temporal edges that define the spatiotemporal AFID trajectory of the behavior. The at least one neural network may comprise at least one graph convolutional network (GCN) for processing the trajectory and classifying the behavior according to the individual performing the behavior.

[0010] In one embodiment, the at least one GCN comprises a nonlocal neural network (NLGCN) having at least one nonlocal neural network block for processing the AFID spatiotemporal trajectories. The at least one nonlocal neural network block may comprise at least one spatial nonlocal neural network block and / or at least one temporal nonlocal neural network block. Optionally, the NLGCN is configured as a multi-stream GCN comprising multiple constituent NLGCNs operative to process a set of data characterized by independent degrees of freedom based on the AFID trajectories. In one embodiment, the output of the multi-stream GCN may include a weighted average of the outputs of each of the constituent GCNSs.

[0011] As an example, in DYNAMIDE configured to identify an individual by the way the individual types, the hand joints, AFIDs, are characterized by degrees of freedom of movement (e.g., distance between different finger joints) that are independent of the degrees of freedom of movement available for the hand bones, AFIDs, that connect the joints. Thus, in one embodiment, DYNAMIDE may include a two-stream 2s-NLGCN multi-stream GCN with two component NLGCNs. One of the two component NLGCNs processes the joint AFIDs, and the other component NLGCN processes the bone AFIDs. In one embodiment, the joint NLGCN comprises at least one learnable "adaptive" adjacency matrix that is substantially data driven to reduce the number of a priori constraints that may be used to construct the 2s-NLGCN. The 2s-NLGCN with adaptive adjacency matrix according to one embodiment of the present disclosure may be referred to as an adaptive 2s-NLGCN (2s-ANLGCN). The output of the 2s-NLGCN or the joint and bone NLGCNs in the 2s-ANLGCN of the typing DYNAMIDE may be fused to identify the individual.

[0012] According to one embodiment, identifying a particular individual occurs in real-time, where real-time identification according to one embodiment refers to identifying an individual while the individual is performing an action or within a time frame in which the quality of experience (QoE) of the service in which the identification occurs is not substantially degraded by the identification process.

[0013] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.

[0014] BRIEF DESCRIPTION OF THE DRAWINGS Non-limiting examples of embodiments of the present invention are described below with reference to the drawings attached hereto, listed after this paragraph. Identical features that appear in more than one figure are generally labeled with the same label in all figures in which the feature appears. In the figures, a label that labels an icon representing a given feature of an embodiment of the present invention may be used to refer to the given feature. Dimensions of features shown in the figures are selected for convenience and clarity of presentation and are not necessarily drawn to scale.

[0015] FIG. 1 is a flow diagram illustrating a process by which DYNAMIDE may process a sequence of video frames of a person performing an action to identify the person, according to an embodiment of the present disclosure.

[0016] FIG. 2 is a diagram illustrating a schematic of a DYNAMIDE system that processes a sequence of video frames of a person typing on an automated teller machine (ATM) keypad to identify the person, according to an embodiment of the present disclosure.

[0017] FIG. 3A is a schematic diagram illustrating a hand image and hand AFID that may be used by DYNAMIDE to identify a person typing on a keyboard, according to an embodiment of the present disclosure.

[0018] FIG. 3B is a schematic diagram of an S-graph, which is a spatial graph that models the hand shown in FIG. 3A, according to an embodiment of the present disclosure.

[0019] FIG. 4A is a schematic diagram illustrating a magnified image of one video frame of a series of video frames captured for a person engaged in typing at the ATM shown in FIG. 2, according to an embodiment of the present disclosure.

[0020] FIG. 4B is a diagram that illustrates a schematic of an S-graph that models the hand imaged in the video frame shown in FIG. 4A, according to an embodiment of the present disclosure.

[0021] FIG. 5A is a schematic diagram illustrating a magnified image of the sequence of video frames shown in FIG. 2, according to an embodiment of the present disclosure.

[0022] FIG. 5B is a schematic diagram illustrating a spatio-temporal graph ST-graph corresponding to images of the sequence of video frames shown in FIG. 5A, according to an embodiment of the present disclosure.

[0023] FIG. 6A is a schematic diagram illustrating a spatiotemporal feature tensor comprising data associated with the nodes of the ST-graph shown in FIG. 5B, according to one embodiment of the present disclosure.

[0024] FIG. 6B illustrates a schema of a non-local neural network that DYNAMIDE may use to process the data in the tensor shown in FIG. 6A, according to an embodiment of the present disclosure.

[0025] Detailed Description In the discussion, unless otherwise stated, adjectives such as "substantially" and "about" modifying the relationship of a state or characteristic of a feature or features of an embodiment of the present disclosure are understood to mean that the state or characteristic is defined within an acceptable tolerance for the process of the desired embodiment of the specification. Whenever a general term in the present disclosure is described by reference to an example or list of examples, the example or examples mentioned are intended to be non-limiting illustrations of the general term. Also, the general term is not intended to be limited to the specific example or examples mentioned. The phrase "in an embodiment" is used to introduce an example consideration, whether or not related to an allowance such as "may," "optionally," or "for illustrative purposes." However, the phrase does not necessarily introduce a required configuration in possible embodiments of the present disclosure. Unless otherwise stated, the term "or" in the specification and claims is considered to be an inclusive "or" rather than an exclusive one, indicating at least one of the multiple items to be combined, or any combination.

[0026] FIG. 1 shows a high-level flow diagram 20, optionally also referenced by the numeral 20, illustrating a process according to an embodiment of the present disclosure, based on which DYNAMIDE may operate to identify a person in response to actions performed by the person.

[0027] In block 22, DYNAMIDE, according to an embodiment of the present disclosure, optionally acquires a sequence of video frames of persons involved in the activity. DYNAMIDE is configured to process to determine the identity of the persons involved in the activity. In block 24, DYNAMIDE processes the video frames to identify images of body regions of interest (BROIs) in the video frames that image at least one AFID associated with the activity. Identifying the BROIs in the video frames optionally includes determining at least one bounding box in the frames that includes an image of the BROI. In block 26, DYNAMIDE processes each of the bounding boxes determined for the video frames to identify an image of at least one AFID in each of the bounding boxes. Identifying the image of the AFID within the bounding box of the video frames optionally includes associating an "AFID ST-ID" with the image, which is a spatiotemporal ID (ST-ID) that includes an identification label of the AFID. Here, an "AFID ST-ID" is used to label all identified images of the same AFID in a video frame and determine the spatiotemporal coordinates of the images. The spatiotemporal coordinates include a timestamp and at least two spatial coordinates. The timestamp identifies the time, temporal position at which the video frame containing the bounding box in which the AFID is located was acquired relative to the time at which other video frames in the sequence of video frames were acquired. The at least two spatial coordinates correspond to the spatial position of the AFID at the time indicated by the timestamp. Optionally, the AFID ST-ID for a given identified AFID includes a standard deviation (sd) for each spatial coordinate and a probability that the AFID-ID label associated with the AFID ST-ID is correct. The earliest and latest timestamps and extreme spatial coordinates determined for the AFID ST-ID determine the spatiotemporal volume.The spatiotemporal volume is sometimes referred to as the spatiotemporal AFID hull (ST-Hull), which contains the spatiotemporal coordinates of all instances of AFIDs imaged and identified in the sequence of video frames.

[0028] In block 28, DYNAMIDE uses the ST-IDs of the AFIDS to configure the identified instances of AFIDs as nodes of an AFID spatio-temporal graph (ST-graph) connected by spatial and temporal edges. The spatial edges connect the ST-graph nodes. The ST-graph nodes represent imaged instances of AFIDs identified by the same timestamp, i.e., instances of AFIDs imaged in the same video frame, and spatial constraints imposed on the AFIDs by the structure of the person's body. The configuration of nodes connected by spatial edges representing the spatial relationship of instances of AFIDs imaged in the same given frame and at a given time may be referred to as the spatial graph (S-graph) of AFIDs at a given time. The temporal edges connect temporally adjacent nodes in the ST-graph representing images of the same AFID in two successively acquired video frames in the sequence of video frames. The temporal edges represent the elapsed time between two successive timestamps. The ST-graph may be considered to include S-graphs corresponding to AFIDs connected by temporal edges.

[0029] In one embodiment, in block 30, DYNAMIDE processes the AFID ST-graph using an optionally adaptive adaptive non-local graph convolutional neural net, ANLGCN. This allows DYNAMIDE to determine, optionally in real time, which of the multiple people that the ANLGCN is trained to recognize are engaged in or about to engage in an action. In one embodiment, the ANLGCN is configured to span the AFID ST-hull and allow data associated with imaged instances of AFID at any spatiotemporal location within the hull to be weighted by the learned weights. The ANLGCN is also configured to contribute to a convolution by the ANLGCN performed on spatiotemporal locations anywhere else within the hull. Optionally, the NLGCN is configured as a multi-stream GCN with multiple component NLGCNs operative to process a set of AFID data characterized by independent degrees of freedom.

[0030] 2 is a schematic diagram of a DYNAMIDE system 100 according to an embodiment of the present disclosure. The DYNAMIDE system 100 is configured to execute the process illustrated in FIG. 1 to identify people involved in an activity based on how they perform the activity. The DYNAMIDE system 100 may optionally include a cloud-based processing hub 120 and an imaging system 110 having a field of view (FOV) indicated by dashed line 111. As an example, in the figure, the activity involves a person 50 at an ATM 60 typing on a keypad 62.

[0031] The imaging system 110 is operable to provide a video sequence 114 consisting of a number ("N") of 2D and / or 3D video frames 114n of a person 50's hands 52 typing on a keypad 62, where 1≦n≦N. The imaging system 110 is connected to the hub 120 by at least one wired and / or wireless communication channel 113, through which the imaging system 110 transmits captured video frames to the hub. The hub 120 is configured to process the received video frames 114n to identify a person 50, whose hands 52 are imaged in the video frames. The hub comprises and / or has access to data and / or executable instructions and any of a variety of electronic and / or optical physical and / or virtual processors, memories, and / or wired or wireless communication interfaces, which may be required to support the functionality provided by the hub. The data and / or executable instructions are hereinafter also referred to as software. The processor, memory, and / or communication interfaces are also referred to hereinafter as hardware.

[0032] By way of example, the hub 120 comprises software and hardware supporting an object detection module 130, an AFID identifier module 140, and a classifier module 150. The object detector module 130 is operable to detect BROIs within the video frames 114n. The AFID identifier module 140 identifies AFIDs within the detected BROIs and provides an ST-ID for each identified AFID. The classifier module 150 comprises a non-local classifier operable to process the set of ST-IDs as a spatio-temporal graph to identify persons 50.

[0033] In one embodiment, the object BROI detector module 130 comprises a fast object detector, such as a You Look Only Once (YOLO) detector, capable of detecting relevant BROIs in real time. The AFID identifier module 140 may comprise a convolutional pose machine (CPM) for identifying AFIDs in detected BROIs. The classifier module 150 comprises an optionally adaptive non-local graph convolutional network, as described above and discussed below. In FIG. 2, the classifier module 150 is shown diagrammatically to provide an output of probabilities represented by a histogram 152. The histogram gives, for each given person of the plurality of persons, the probability that the given person is the person whose hands 52 are imaged typing in the video frame. DYNAMIDE 100 is trained to recognize that the given person is the person whose hands 52 are imaged typing in the video frame. DYNAMIDE 100 is shown diagrammatically as successfully identifying person 50 as the person whose typing hands 52 are imaged in video frame 114n.

[0034] In one embodiment, the AFIDs used by DYNAMIDE 100 to identify a person's typing are the joints (knuckles and / or wrist) and finger bones (phalanges) of the typing hand. FIG. 3A illustrates an image of a hand 200 with knuckles (also called knuckles) and wrist joints that are optionally used as AFIDs by DYNAMIDE 100 to process video images of a typing hand, according to an embodiment of the present disclosure. The joints have locations on the hand 200 indicated by a plus sign "+" and may be collectively referenced by the wrist label "JH" as shown, and individually distinguished by numeric labels J0, J1, ..., J20. A given phalange that DYNAMIDE 100 may use as an AFID for a typing activity is identified when the given phalange is referenced by an alphanumeric label indicating the two knuckles it connects. For example, in Figure 3A, the finger bone connecting joints J5 and J6 is indicated diagrammatically in Figure 3A by a dashed line labeled B5-6, and phalange B18-19 connects finger joints J18 and J19. The finger bones may be referred to collectively by the label BH.

[0035] FIG. 3B is a schematic diagram of a spatial graph (S-graph 200) that may be used to represent the spatial relationships of AFIDs at a given time, according to an embodiment of the present disclosure. As an example, the spatial graph is shown by the hand 200 at a given time when the hand 200 is imaged. In the spatial S-graph 200, the finger joint AFIDs JH shown in FIG. 3A are represented by nodes that are collectively referenced by the label JN. The nodes JN are individually distinguished by alphanumeric labels JN0, JN1, ..., JN20 that correspond to the homologous finger joints J0, J1, ..., J20 shown in FIG. 3A, respectively. The edges of the S-graph 200 that connect the nodes JN represent the finger bones, i.e., the bones AFIDs that connect the finger joints. As shown in FIG. 3B, the edges may be collectively referenced by the label BE and individually referenced by the reference labels that correspond to the homologous finger bones in the hand 200. For example, edge BE5-6 in FIG. 3B corresponds to bone B5-6 in FIG. 3A.

[0036] 4A illustrates a schematic representation of a zoomed-in image of an nth video frame 114n in an array of video frames 114 (FIG. 2) acquired by imaging system 110 at acquisition time tn and transmitted to DYNAMIDE hub 120 for processing, according to one embodiment of the present disclosure. Video frame 114n images hands 52 typing on keypad 62, as well as features in the environment surrounding the hands that may be located within the FOV 111 (FIG. 2) of imaging system 110. The surrounding features illustrated in FIG. 4A may include, for example, parts of the structure of ATM 60, such as counter 64 and sidewall 66, as well as a mobile phone 55 that person 50 has placed on counter 64.

[0037] As described above, in processing the sequence 114 of video frames 114n, the object detection module 130 may determine a bounding box that locates the image of the hand 52 in the frame as an object with a joint AFID that the AFID detector 140 identifies and that DYNAMIDE 100 uses to identify the person 50. The bounding box determined by the object detector module 130 for the hand 52 in the video frame 114n is indicated by the dashed rectangle 116. The knuckle AFID that the AFID detector 140 detects and identifies in the bounding box 116 is indicated by the generic AFID label JH (FIG. 3A). FIG. 4B illustrates a schematic of a spatial S-graph-52(tn) that models the hand 52 as a graph based on the image of the hand in the video frame 114n acquired at an acquisition time tn. The knuckle nodes in S-Graph-52(tn) are indicated by the appropriate knuckle node label JN0, JN1, ..., JN20, and may be supplemented with an argument indicating the acquisition time tn associated with the S-Graph-52(tn) to which the node belongs. For example, nodes JN0, JN1, ..., JN20 in S-Graph-52(tn) may be referenced as JN0(tn), JN1(tn), ..., JN20(tn).

[0038] FIG. 5A illustrates a schematic representation of a zoomed-in image of the video sequence 114 shown in FIG. 2, including video frames 1141, 1142, 1143, ..., 114N that image a hand 52 typing on an ATM 60 at respective times t1, t2, t3, ..., tN. FIG. 5B illustrates a schematic representation of an ST-graph 52 that models the spatiotemporal progression of a typing action based on images of the hand 52 in video frames 1141-114N, according to an embodiment of the present disclosure. The ST-graph 52 includes spatial S-graphs-52(tn) that correspond to the images of the hand 52 in the video frames 1141, ..., 114N, where 1≦n≦N. Homologous nodes JN in adjacent S-graphs, S-graph-52(tn) and S-graph-52(tn+1), are connected by a time edge that represents the elapsed time between the respective acquisition times tn and tn+1. All the time edges between adjacent S-graphs-52(tn) and S-graphs-52(tn+1) have the same time length and are labeled TEn, n+1. Some of the time edges in Figure 5B are labeled with their respective labels.

[0039] The node data associated with the ST-graph-52 provides a set of spatio-temporal input features that the classifier module 150 of the DYNAMIDE hub 120 processes to determine the identity of the person 50 typing on the keypad 62 of the ATM 60. The set of input features may be modeled as an input spatio-temporal feature tensor 300, as shown diagrammatically in FIG. 6A, with AFID, time, and channel axes indicating a position in the tensor by row, column, and depth. In the ST-graph-52, the AFID axis is calibrated with a node number indicating a particular joint in the hand 52, and the time axis is calibrated with a sequential frame number or frame acquisition time. As an example, the channel axis of the spatio-temporal feature tensor 300 diagrammatically shows four channels, although it should be noted that a spatio-temporal feature tensor according to an embodiment may have more or less than four channels. For example, an entry along the channel axis corresponding to a given node and a given time shown along the AFID and time axes, respectively, may provide two or three spatial coordinates that correspond to a given node at a given time and determine a spatial location. The channel entry may also provide an error estimate of the accuracy of the coordinates and the probability that a given node will be correctly identified.

[0040] In one embodiment, the classifier module 150 according to one embodiment of the present disclosure may have a classifier including at least one non-local graph convolutional net (NLGCN) for processing the data in the tensor 300 to provide the identity of the person 50. Optionally, the at least one NLGCN comprises at least one adaptive ANLGCN including an adaptive adjacency matrix in addition to a non-local GCN layer. The adaptive adjacency matrix operates to improve the classifier recognition of spatio-temporal motions at the joints of the hand relative to each other. The spatio-temporal motions are not affected by spatial structures and are specific to the way the person performs typing.

[0041] As an example, FIG. 6B illustrates a schema of a classifier 320 that the DYNAMIDE hub 120 may use to process data in tensor 300, according to one embodiment of the present disclosure. Classifier 320 optionally includes convolutional neural network blocks 322, 324, and 326 that feed data forward to a fully connected net FCN 328 that provides a probability for each of a number of people as to whether or not the person is the person whose hands 52 were imaged in the video sequence 114 (FIG. 2) typing on the keypad 62. Block 322 optionally includes a GCN that feeds forward data to a time convolutional network (TCN). Block 324 includes an ANL-GCN that feeds data forward to the TCN, and block 326 includes a GCN that feeds data forward to the ANL-TCN.

[0042] Thus, according to one embodiment of the present disclosure, there is provided a method for identifying a person, the method comprising: acquiring spatiotemporal data for each of a plurality of anatomical landmarks associated with an activity involving a person, the anatomical landmarks providing data determining at least one spatiotemporal trajectory of the anatomical landmarks during the activity; modeling the acquired spatiotemporal data as a space-time graph (ST-graph); and processing the ST-graph with at least one non-local graph convolutional neural network (NLGCN) to provide an identity corresponding to the person. Optionally, the at least one NLGCN comprises at least one adaptive NLGCN (ANLGCN) including an adaptive adjacency matrix that is learned in response to data relating to anatomical landmarks of the plurality of anatomical landmarks, the anatomical landmarks of the plurality of anatomical landmarks being not determined solely by the body structure of the person. Additionally or alternatively, processing the ST-graph comprises segmenting the plurality of anatomical landmarks into a plurality of sets of anatomical landmarks, each set being characterized by a configuration with a different degree of freedom of movement. Optionally, the method includes modeling the acquired spatiotemporal data associated with the anatomical landmarks in each set as an ST-graph. The processing step includes processing the ST-graph modeled for each set in the plurality of sets of anatomical landmarks with an NLGCN of the at least one NLGCN to determine data indicative of the person's identity, the determining may be independent of processing other sets of the plurality of sets. The method optionally includes fusing the determined data from all the sets to provide the identity for the person.

[0043] In one embodiment, acquiring the spatiotemporal data includes acquiring a series of video frames imaging the person engaged in the behavior, each video frame including an image of at least one body region of interest (BROI) imaging an anatomical landmark of the plurality of anatomical landmarks. Optionally, the method includes processing the video frames to detect the at least one BROI in each video frame. Additionally or alternatively, the method optionally includes identifying an image of an anatomical landmark of the plurality of anatomical landmarks in each of the at least one detected BROI. Optionally, the method includes processing the images of the identified anatomical landmarks to determine the data defining the spatiotemporal trajectory.

[0044] In one embodiment, the plurality of anatomical landmarks includes joints. Optionally, the plurality of anatomical landmarks includes bones connecting the joints. Additionally or alternatively, the joints include finger joints. Optionally, the actions include a series of finger movements. The finger movements may include movements involved in operating a keyboard.

[0045] In one embodiment, the joint comprises a large appendage joint. Optionally, the activity is a sport. Optionally, the sport is soccer. Optionally, the sport is golf.

[0046] In one embodiment, the plurality of anatomical landmarks includes facial landmarks. Optionally, the facial landmarks include facial landmarks whose movements are used to define Facial Action Coding System (FACS) Action Units (AUs) used to classify facial expressions and micro-expressions. In one embodiment, the plurality of anatomical landmarks includes detailed paired features of fingerprints of multiple fingers of a hand.

[0047] Further, in accordance with an embodiment of the present disclosure, there is provided a system for identifying a person, comprising: an imaging system operable to capture footage having video frames capturing a person engaged in an activity, and software operable to process the video frames in accordance with any of the preceding claims to provide an identity corresponding to the person.

[0048] The descriptions of the embodiments of the invention in this application are provided by way of example and are not intended to limit the scope of the invention. The described embodiments include different features, not all of which are required in all embodiments. Some embodiments utilize only some of the features or possible combinations of the features. Variations of the described embodiments of the invention, and embodiments including different combinations of the features set forth in the described embodiments, will occur to those skilled in the art. The scope of the invention is limited only by the claims. [Brief description of the drawings]

[0049] [Figure 1] A flow diagram illustrating a process by which DYNAMIDE may process a sequence of video frames of a person performing an action to identify the person, according to an embodiment of the present disclosure. [Diagram 2] FIG. 1 is a schematic diagram of a DYNAMIDE system for processing a sequence of video frames of a person typing on an automated teller machine (ATM) keypad to identify the person, according to an embodiment of the present disclosure. [Figure 3A] FIG. 3A is a schematic diagram illustrating a hand image and hand AFID that may be used by DYNAMIDE to identify a person typing on a keyboard, according to an embodiment of the present disclosure. [Figure 3B] FIG. 3B is a schematic diagram of an S-graph, which is a spatial graph that models the hand shown in FIG. 3A, according to an embodiment of the present disclosure. [Figure 4A]FIG. 4A is a schematic diagram illustrating a magnified image of one video frame of a series of video frames captured for a person engaged in typing at the ATM shown in FIG. 2, according to an embodiment of the present disclosure. [Figure 4B] FIG. 4B is a diagram that illustrates a schematic of an S-graph that models the hand imaged in the video frame shown in FIG. 4A, according to an embodiment of the present disclosure. [Figure 5A] FIG. 5A is a schematic diagram illustrating a magnified image of the sequence of video frames shown in FIG. 2, according to an embodiment of the present disclosure. [Figure 5B] FIG. 5B is a schematic diagram illustrating a spatio-temporal graph ST-graph corresponding to images of the sequence of video frames shown in FIG. 5A, according to an embodiment of the present disclosure. [Figure 6A] FIG. 6A is a schematic diagram illustrating a spatiotemporal feature tensor comprising data associated with the nodes of the ST-graph shown in FIG. 5B, according to one embodiment of the present disclosure. [Figure 6B] FIG. 6B illustrates a schema of a non-local neural network that DYNAMIDE may use to process the data in the tensor shown in FIG. 6A, according to an embodiment of the present disclosure.

Claims

1. 1. A method for identifying a person, comprising: storing data and executable instructions in a memory; executing the executable instructions in a processor to perform the steps of: The process comprises: acquiring spatiotemporal data for each of a plurality of anatomical landmarks associated with an activity involving a human being, the data providing data defining a spatiotemporal trajectory of at least one of the anatomical landmarks during the activity; modelling the acquired spatio-temporal data as a spatio-temporal graph (ST-graph); processing the ST-graph with at least one non-local graph convolutional neural network (NLGCN) by segmenting the plurality of anatomical landmarks into a plurality of sets of anatomical landmarks; outputting an identity of the person based on the processing of the ST-graph; Including, each of the plurality of sets being characterized by a respective one of a plurality of degrees of freedom of motion, the degrees of freedom of motion being independent of one another; method.

2. 2. The method of claim 1, wherein the at least one NLGCN comprises at least one adaptive NLGCN (ANLGCN) including an adaptive adjacency matrix that is learned in response to data regarding anatomical landmarks of the plurality of anatomical landmarks, and the anatomical landmarks of the plurality of anatomical landmarks are not determined solely by the person's body structure.

3. The method of claim 1 , comprising modeling the acquired spatio-temporal data associated with the anatomical landmarks in each set as an ST-graph.

4. 4. The method of claim 3, wherein the processing step includes processing the ST-graph modeled for each set in the plurality of sets of anatomical landmarks with an NLGCN of the at least one NLGCN to determine data indicative of the person's identity, the determining being independent of processing other sets of the plurality of sets.

5. The method of claim 4 , comprising fusing the determined data from all the sets to provide the identity for the person.

6. 2. The method of claim 1, wherein acquiring the spatiotemporal data includes acquiring a series of video frames imaging the person engaged in the activity, each video frame including an image of at least one body region of interest (BROI) imaging an anatomical landmark of the plurality of anatomical landmarks.

7. The method of claim 6 , comprising processing the video frames to detect the at least one BROI in each video frame.

8. The method of claim 6 , further comprising identifying, in each of the at least one detected BROI, an image of an anatomical landmark of the plurality of anatomical landmarks.

9. The method of claim 8 , comprising processing images of the identified anatomical landmarks to determine the data defining the spatiotemporal trajectory.

10. The method of claim 8 , wherein the plurality of anatomical landmarks includes joints.

11. The method of claim 10 , wherein the plurality of anatomical landmarks includes bones connecting the joints.

12. The method of claim 10 , wherein the joints include finger joints.

13. The method of claim 12 , wherein the actions include a sequence of finger movements.

14. The method of claim 13 , wherein the finger movements include movements involved in operating a keyboard.

15. The method of claim 10 , wherein the joint comprises a large appendage joint.

16. The method of claim 15 , wherein the activity is a sport.

17. The method of claim 16, wherein the sport is soccer.

18. The method of claim 16 , wherein the sport is golf.

19. 1. A method for identifying a person, comprising: storing data and executable instructions in a memory; executing the executable instructions in a processor to perform the steps of: The process comprises: acquiring spatiotemporal data for each of a plurality of anatomical landmarks associated with an activity involving a human being, the anatomical landmarks providing data determining a spatiotemporal trajectory of at least one of the anatomical landmarks during the activity; modelling the acquired spatio-temporal data as a spatio-temporal graph (ST-graph); processing the ST-graph with at least one non-local graph convolutional neural network (NLGCN) including a fully connected network (FCN) configured to generate probability histograms for a plurality of persons associated with the anatomical landmarks; outputting an identity of the person based on the probability histogram; Including, The method, wherein the plurality of anatomical landmarks includes facial landmarks.

20. 20. The method of claim 19, wherein the facial landmarks include facial landmarks whose movements are used to define Facial Action Coding System (FACS) action units (AUs) used to classify facial expressions and micro-expressions.

21. the plurality of anatomical landmarks including detailed paired features of fingerprints of a plurality of fingers of a hand; 20. The method of claim 19.

Images