Anomaly detector using complementary classifiers to detect anomalies

Abstract

Embodiments of the present disclosure disclose an anomaly detector for detecting anomalies in a sequence of postures of a person performing an activity. The anomaly detector comprises: an input interface configured to accept input data indicative of a distribution of the sequence of postures; a memory configured to store a discriminative one-class classifier of a pair of complementary classifiers having a normal distribution bounding the sequence of postures in a Reproducing Kernel Hilbert Space (RKHS); a processor configured to embed the input data into an element of the RKHS and to classify the embedded data using the discriminative one-class classifier; and an output interface configured to render a classification result.

Description

TECHNICAL FIELD

[0001] The present disclosure relates generally to anomaly detection, and more particularly, to detecting anomalies in a sequence of human poses using an anomaly detector. BACKGROUND

[0002] Generally, a human pose indicates various postures that can be assumed by a human body in daily, special, or celebratory situations. Human poses can vary greatly depending on environmental factors, lifestyle factors, and the like. A few common examples of human poses include standing pose, sitting pose, walking pose, sleeping pose, running pose, and the like. Additionally, human poses such as these can provide insight into the type of human activity being performed by different individuals. For example, a dancer’s dance pose can indicate a dance activity. A chef’s cooking pose can indicate a cooking activity, and a driver’s driving pose can indicate a driving activity. Thus, a human’s human pose indicates one or more human activities. Conversely, an anomaly in a human pose can indicate an anomalous human activity.

[0003] Nowadays, with technological advancements and developments, it can be desirable to detect anomalies in the poses of humans performing activities for many applications. For example, anomaly detection in human poses can be useful in safety applications such as driver assistance applications that check the attentiveness of drivers operating vehicles, airport security, and health-related applications that assist in the safety of people being cared for, and the like.

[0004] However, anomaly detection in human poses can be a challenging task due to factors related to training data or labeled data corresponding to human anomalous poses. Further, due to the non-linearity and complexity of human poses, it can be difficult to obtain training data for human anomalous poses. The complexity and non-linearity of human poses can be caused by variations in human poses from one individual to another. Further, manually annotating labeled data can be a monotonous and challenging task.

[0005] Therefore, there is a need for a solution to overcome the aforementioned challenges of detecting anomalies in human poses in an efficient and feasible manner. SUMMARY

[0006] It is an object of some embodiments to employ an anomaly detector to detect anomalous poses in a sequence of human poses. It is another object of some embodiments to perform classification of anomalous data corresponding to anomalous human poses from normal data corresponding to normal human poses in a sequence of human poses.

[0007] Some implementations are based on the recognition that the unavailability of training data corresponding to abnormal human poses can be addressed by a discriminative one-class classifier. The discriminative one-class classifier allows a model to be trained using normal poses of a person performing an activity and allows the trained model to be used to detect poses that do not correspond to the normal poses of the person performing the activity as abnormal poses. To this end, a single discriminative classifier can be trained to detect abnormalities of the person performing the activity.

[0008] Some implementations are based on the recognition that the nonlinearity and complexity of human poses can be addressed by providing a pair of complementary classifiers instead of providing a single discriminative classifier. To this end, the discriminative one-class classifier includes a pair of complementary classifiers. Each discriminative classifier in the pair of complementary classifiers can be trained to bound a space corresponding to the normal poses of the person performing the activity from different directions. To this end, normal data can be bounded within the boundaries of the pair of complementary classifiers. For example, one half space (e.g., positive half space) of the space corresponding to the normal poses of the person performing the activity can be bounded by one discriminative classifier in the pair of complementary classifiers and the other half space (e.g., negative half space) of the space can be bounded by the other discriminative classifier in the pair of complementary classifiers.

[0009] Further, the pair of complementary classifiers can represent the complexity of the normal pose distribution of different human activities. For example, the pair of complementary classifiers provide better representation flexibility and more degrees of freedom in characterizing the data distribution of the normal poses of different human activities while supporting a single type of one-class data. However, utilizing a pair of complementary classifiers instead of a single discriminative classifier addresses some problems. However, it brings many other problems. Specifically, learning a pair of complementary classifiers on the non-linear training data of human poses can be a non-convex optimization problem.

[0010] Some implementations are based on the recognition that the non-convex optimization problem can be addressed with various optimization techniques, e.g., using Riemannian conjugate gradient optimization on Stiefel manifolds. In mathematics, a Stiefel manifold is the set of all orthogonal k-frames. In other words, a Stiefel manifold is the space of orthogonal (linear) hyperplanes. However, it is difficult to apply Stiefel manifolds on arbitrarily shaped boundaries of discriminative classifiers resulting from the nonlinearity of human poses.

[0011] To this end, it is an objective of some implementations to learn a pair of complementary classifiers in a Hilbert space, such as a Reproducing Kernel Hilbert Space (RKHS). The RKHS provides potentially infinite linear dimensions, which ensures that the boundary of one or more straight lines or hyperplanes of the pair of complementary classifiers is linear. Examples of linear boundaries can include a pair of straight line segments (e.g., hyperplanes), a pair of straight lines and orthogonal (standard orthogonal) frames, a pair of standard and non-orthogonal frames (e.g., hyperspheres).

[0012] Some embodiments can use kernel embedding to project the posture distribution into the RKHS. In some other embodiments, different kernels can be used based on the data distribution of the postures to be classified. For example, some embodiments can use a histogram cross kernel for posture distributions captured by histograms. Another embodiment can use a Chi-sq kernel for histogram data, a linear kernel or a radial basis function (RBF) kernel can be used for data in a spherical distribution, two complementary RBF kernels can be used for data in a ring distribution. Another embodiment can use a tree kernel for hierarchical data.

[0013] In various embodiments, the kernel embedding is intermediate and unobserved. The kernel embedding remains internal to the optimization and / or classification during training as well as during the inference phase of the abnormal posture classification. As a result, this kernel-based classification with a pair of complementary classifiers that bounds the distribution space of normal human postures allows to reduce the computational complexity of the anomaly detection of human activities in the absence of training data of abnormal postures of such activities.

[0014] In some embodiments, the anomaly detection of human postures can be implemented in an application such as a vehicle driver assistance system that provides safety assurance to a driver of a vehicle as well as one or more passengers of the vehicle. To this end, one or more image capturing devices installed in the vehicle can be used to capture a sequence of image frames comprising one or more images of one or more passengers in the vehicle. The sequence of image frames can be provided as input data for the anomaly detection of the postures of the one or more passengers.

[0015] Accordingly, one embodiment discloses an anomaly detector for detecting anomalies in a sequence of postures of a human performing an activity. The anomaly detector comprises an input interface, a memory, a processor, and an output interface. The input interface is configured to accept input data indicative of a distribution of the sequence of postures. The memory is configured to store a discriminative one-class classifier having a pair of complementary classifiers that bounds a normal distribution of the sequence of postures in a Reproducing Kernel Hilbert Space (RKHS). The processor is configured to embed the input data into an element of the RKHS and to classify the embedded data using the discriminative one-class classifier. The output interface is configured to render a classification result.

[0016] Another embodiment discloses a method for detecting anomalies in a sequence of postures of a human performing an activity. The method comprises the step of accepting input data indicative of a distribution of the sequence of postures. The method comprises the step of embedding the input data into an element of a Reproducing Kernel Hilbert Space (RKHS). The method comprises the step of classifying the embedded data using a discriminative one-class classifier. The discriminative one-class classifier comprises a pair of complementary classifiers that bounds a normal distribution of the sequence of postures in the RKHS. The method further comprises the step of rendering a classification result.

[0017] Additional features and advantages will become more readily apparent from the following detailed description when taken in conjunction with the accompanying drawings.

[0018] The present disclosure is further described in the detailed description that follows, in reference to the noted plurality of drawings wherein like numerals represent like elements throughout the several views. The described drawings are not necessarily to scale, emphasis instead being placed upon illustrating the principles of the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0019] [ Figure 1 ] Figure 1 A schematic diagram illustrating depicting pose sequence anomaly detection using an anomaly detector according to some embodiments of the present disclosure is shown.

[0020] [ Figure 2 ] Figure 2 A block diagram of an anomaly detector according to some embodiments of the present disclosure is shown.

[0021] [ Figure 3A ] Figure 3A A graphical representation depicting a pair of complementary classifiers discriminating a one-class classifier according to some example embodiments of the present disclosure is shown.

[0022] [ Figure 3B ] Figure 3B A graphical representation depicting a pair of complementary classifiers discriminating a one-class classifier according to some other example embodiments of the present disclosure is shown.

[0023] [ Figure 3C ] Figure 3C A graphical representation depicting a pair of complementary classifiers discriminating a one-class classifier according to some other example embodiments of the present disclosure is shown.

[0024] [ Figure 4 ] Figure 4 A pipeline depicting training of a one-class classifier discriminating according to some embodiments of the present disclosure is shown.

[0025] [ Figure 5 ] Figure 5 A pipeline depicting detection of anomalies in human poses by an anomaly detector based on a trained one-class classifier discriminating according to some example embodiments of the present disclosure is shown.

[0026] [ Figure 6 ] Figure 6 A graphical representation depicting classification results corresponding to anomalies in detected pose sequences according to some example embodiments is shown.

[0027] [ Figure 7 ] Figure 7An environment representation depicting anomaly detection using an anomaly detector in a vehicle driver assistance system is shown, in accordance with some example embodiments.

[0028] [ Figure 8 ] Figure 8 A block diagram depicting detection of anomalous poses in a sequence of image frames by an anomaly detector is shown, in accordance with some example embodiments.

[0029] [ Figure 9 ] Figure 9 A schematic diagram depicting a normalized pose sequence and a normalized histogram is shown, in accordance with some example embodiments of the present disclosure.

[0030] [ Figure 10 ] Figure 10 A method flow for detecting anomalies in a sequence of poses of a person performing an activity is shown, in accordance with some example embodiments of the present disclosure.

[0031] [ Figure 11A ] Figure 11A A use case implementation of an anomaly detector is shown, in accordance with some example embodiments of the present disclosure.

[0032] [ Figure 11B ] Figure 11B A use case implementation of an anomaly detector is shown, in accordance with some other example embodiments of the present disclosure.

[0033] [ Figure 12 ] Figure 12 A general block diagram of an anomaly detector is shown, in accordance with some example embodiments of the present disclosure. DETAILED DESCRIPTION

[0034] While the above-described diagrams set forth presently disclosed embodiments, other embodiments can also be contemplated as discussed in the Discussion. The present disclosure presents illustrative embodiments as representations rather than limitations. Numerous modifications and embodiments can be devised by those skilled in the art, which will fall within the scope and spirit of the presently disclosed embodiments.

[0035] In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the present disclosure. It will be apparent, however, to one skilled in the art that the present disclosure can be practiced without these specific details. In other instances, devices and methods are shown in block diagram form in order to avoid obscuring the present disclosure.

[0036] As used in the specification and claims, the terms "for example," "e.g.," and "such as," and the verbs "comprising," "having," "including," and their other verb forms, when used in conjunction with a listing of one or more components or other items, are each to be construed as open-ended, meaning that the listing is not to be considered as excluding other, additional components or items. The term "based on" means at least partially based on. Furthermore, it is to be understood that the phraseology or terminology employed herein is for the purpose of description and not of limitation. Any use of heading title or section title is meant for convenience only and is not to be used against a meaning to the description or the claims.

[0037] SUMMARY

[0038] The proposed anomaly detector is capable of detecting anomalies in human poses. Due to the variations in human poses, human poses can constitute complex and non-linear data. To this end, the goal of the anomaly detector is to detect anomalies in human poses from the distribution of pose sequences. In some implementations, the anomalies are classified from normal data, where the normal data corresponds to a normal (i.e., non-anomalous uniclass) distribution of pose sequences based on a classifier such as a discriminative one-class classifier. The discriminative one-class classifier includes a pair of complementary classifiers that bound a normal distribution of pose sequences in a reproducing kernel Hilbert space. To this end, the pair of complementary classifiers can be optimized, where the distance between the pair of complementary classifiers is minimized and the margin of each of the pair of complementary classifiers from the normal data is minimized. The optimization of the pair of complementary classifiers allows for bounding the normal distribution of pose sequences from different directions in an accurate and compact manner. Thus, the discriminative one-class classifier detects outliers as anomalies in human poses from the bounded normal distribution of pose sequences. The anomaly detection in human poses can be implemented in different applications, such as vehicle driver assistance systems, security monitoring systems, etc.

[0039] Figure 1 A schematic diagram 100 depicting anomaly detection in a sequence of poses 102 using an anomaly detector 106 according to some implementations of the present disclosure is shown. In an example implementation, the sequence of poses 102 can correspond to a set of image frames 104 of a person performing an activity. The set of image frames 104 can be obtained from an image capturing device, such as a video camera Figure 1(Not shown in the image). The pose sequence 102 may include different human poses, which may be complex and non-linear data. Furthermore, some poses in the pose sequence 102 may correspond to normal poses. Normal poses may include normal data from the pose sequence 102. In some embodiments, this normal data of normal poses can be used to classify outliers in the pose sequence 102 as anomalous data. Anomalous data may correspond to anomalous poses in the pose sequence 102. Therefore, normal data and anomalous data in the pose sequence 102 can be distinguished. For this purpose, an arbitrary-shaped boundary of the discriminant class can be used to distinguish normal data and anomalous data. However, the complexity and non-linearity in the pose sequence 102 may affect the arbitrary-shaped boundary of the discriminant classifier.

[0040] Therefore, the attitude sequence 102 is provided to the anomaly detector 106 for anomaly detection in the attitude sequence 102. In some embodiments, the distribution of the attitude sequence 102 is provided as input data to the anomaly detector 106 to generate anomalous attitudes in the attitude sequence 102 as output 108. (Refer to...) Figure 2 Anomaly detector 106 is further described.

[0041] Figure 2 A block diagram of an anomaly detector 106 according to some embodiments of the present disclosure is shown. The anomaly detector 106 includes an input interface 202, a memory 204, a processor 208, and an output interface 210. The input interface 202 is configured to accept input data. The input data indicates an attitude sequence (…). Figure 1 The distribution of the posture sequence 102). Memory 204 is configured to store a discriminative single-class classifier 206. The discriminative single-class classifier 206 includes a pair of complementary classifiers for defining the normal distribution of the posture sequence in the reproducing kernel Hilbert space (RKHS). Alternatively or additionally, the discriminative single-class classifier 206 may include multiple complementary classifiers for defining the normal distribution of the posture sequence. As used herein, the normal distribution of the posture sequence corresponds to the distribution of a normal posture sequence of a person's posture during an activity.

[0042] The processor 208 is configured to embed the input data into the RKHS. Due to the complexity and nonlinearity of the input data, the input data is embedded into the RKHS. To this end, the pose sequence 102 can be embedded into a kernel for projecting the normal distribution of the pose sequence into the RKHS. The RKHS includes an infinite dimensional linear space that allows the non-linear input data to be projected into a linear space. The infinite dimensional linear space of the RKHS allows the pair of complementary classifiers to learn better as compared to other low dimensional spaces. For example, the infinite dimensional linear space of the RKHS allows the pair of complementary classifiers to bound the non-linear input data from different directions. To this end, the RKHS can address the complexity and nonlinearity of the input data. The pair of complementary classifiers can be non-linear to bound the non-linear input data. However, the individual boundaries of the pair of complementary classifiers can be linear in the RKHS. In different implementations, different kernels can be used depending on the distribution of the data to be classified. For example, in some implementations, a histogram cross kernel can be used for the distribution of the poses captured by the histogram. Likewise, a chi-square kernel can be used for the histogram data, a linear or RBF kernel can be used for the spherical data, and a tree kernel can be used for the hierarchical data.

[0043] In some implementations, each of the pair of complementary classifiers in the discriminative one-class classifier 206 bounds the normal data corresponding to the normal distribution of the pose sequence (i.e., the pose sequence of normal human poses) from different directions. For example, one of the pair of complementary classifiers can classify the normal data into a positive half-space in the RKHS, and the other classifier of the pair of complementary classifiers can classify the normal data into a negative half-space in the RKHS. Thus, the normal data can be bounded from complementary directions using the pair of complementary classifiers. Some non-limiting examples of the pair of complementary classifiers can include, but are not limited to, a pair of hyperplanes (e.g., hyperplanes), a pair of standard orthogonal frames (e.g., a line and an orthogonal matrix), a pair of standard but non-orthogonal frames, and a pair of non-standard orthogonal matrices (e.g., hyperspheres), which are illustrated in Figure 3A , Figure 3B and Figure 3C .

[0044] Further, the processor 208 is configured to classify the embedded data using the discriminative one-class classifier 206. The classified embedded data can be provided to the output interface 210 to render the classification result. The classification result can correspond to the abnormal pose detected in the pose sequence 102. In some example implementations, the processor 208 is further configured to output a notification to the person performing the activity based on the detected abnormality. The notification can be provided to the person via the output interface 210.

[0045] Figure 3A A graphical representation 300A depicting a pair of complementary classifiers according to some example implementations of the present disclosure is shown. In some example implementations, the pair of complementary classifiers corresponds to a pair of hyperplanes 302A and 302B, as illustrated inFigure 3A As shown, each of the pair of hyperplanes 302A and 302B can include a two- dimensional (2D) plane or a line. As shown, the pair of hyperplanes 302A and 302B can bound a normal distribution of the sequence of poses 304. The normal distribution of the sequence of poses 304 corresponds to normal sequences of poses in the sequence of poses 102. Figure 3A

[0046] Figure 3B A graphical representation 300B depicting a pair of complementary classifiers according to some other example implementations of the disclosure is shown. In some example implementations, the pair of complementary classifiers of the discriminative one-class classifier 206 can correspond to a pair of standard orthogonal frames 306A and 306B, as shown. Figure 3B As shown, each of the pair of standard orthogonal frames 306A and 306B can include two or more linear hyperplanes that are orthogonal to each other. The two or more linear hyperplanes are grouped to form each of the pair of standard orthogonal frames 306A and 306B while providing a piecewise linear boundary for bounding the normal distribution of the sequence of poses 304. The piecewise linear boundary also enables the pair of standard orthogonal frames 306A and 306B to perform in an efficient manner while inferring classification of abnormal data from normal data in the normalized distribution of the sequence of poses 304.

[0047] In some cases, the normal distribution of the sequence of poses 304 can be distributed in a spherical manner. In these cases, due to the spherical shape or torus shape, the pair of standard orthogonal frames 306A and 306B or the pair of hyperplanes 302A and 302B can not bound one or more data points in the normal distribution of the sequence of poses 304. To this end, the spherical distribution of the normal distribution of the sequence of poses 304 can be bounded by a pair of hyperspheres, as shown. Figure 3C

[0048] Figure 3C A graphical representation 300C depicting a pair of complementary classifiers according to some other example implementations of the disclosure is shown. In some example implementations, the pair of complementary classifiers of the discriminative one-class classifier 206 can correspond to a pair of hyperspheres 308A and 308B that bound the normal distribution of the sequence of poses 304 in a spherical shape within a spherical region.

[0049] Accordingly, the discriminative one-class classifier 206 bounds the normal distribution of the sequence of poses 304 in different distribution shapes within the boundaries of the pair of complementary classifiers, such as the pair of hyperplanes 302A and 302B, the pair of standard orthogonal frames 306A and 306B, and the pair of hyperspheres 308A and 308B. To this end, the discriminative one-class classifier 206 is trained for bounding the normal distribution of the sequence of poses 304, which is described next with reference to Figure 4

[0050] ​​​Figure 4 A pipeline 400 depicting training of the discriminative one-class classifier 206 in accordance with some embodiments of the present disclosure is shown. The pipeline 400 begins at step 402. At step 402, a sequence of poses 102 is provided as input to the discriminative one-class classifier 206. The sequence of poses 102 can correspond to normal human poses. For example, the sequence of poses 102 can correspond to a predefined dataset of normal human poses. At step 404, the sequence of poses 102 is preprocessed to extract normal sequences of poses. The extracted normal sequences of poses are provided to a classification model of the discriminative one-class classifier 206.

[0051] At step 406, pose features are extracted from the extracted normal sequences of poses based on the classification model. In some example embodiments, the classification model can include a bag-of-words (BoW) model, a convolutional neural network (CNN) model, a deep learning model, and the like. In some example embodiments, the BoW model can be trained to embed the extracted sequences of poses into a fixed-dimensional histogram vector to facilitate extraction of the pose features. In some other example embodiments, the CNN model can be trained as an autoencoder to encode the extracted sequences of poses into a vector to facilitate extraction of the pose features. In some cases, the CNN model can be trained on a dataset corresponding to skeleton-based human poses to facilitate extraction of the pose features. These cases of training the CNN model can occur in applications such as skeleton-based action recognition.

[0052] At step 408, the discriminative one-class classifier 206 is trained based on the extracted pose features. To this end, the extracted pose features are provided to a pair of complementary classifiers (e.g., a pair of hyperplanes 302A and 302B, a pair of standard orthogonal frames 304A and 304B, and a pair of hyperplanes 306A and 306B) to learn a distribution model of the normalized sequences of poses (e.g., the normalized sequence of poses 304). This enables the discriminative one-class classifier 206 to estimate a data density of the normalized distribution of the sequence of poses 304.

[0053] In some embodiments, the discriminative one-class classifier 206 can estimate the data density of the normal distribution of the sequence of poses based on a min-max optimization technique. The min-max optimization technique is used to minimize a distance between individual classifiers of the pair of complementary classifiers and maximize a margin of individual classifiers of the pair of complementary classifiers for a maximum encapsulation of the normal distribution of the sequence of poses. In some example embodiments, the min-max optimization technique corresponds to a Riemannian conjugate gradient algorithm.

[0054] To this end, some embodiments aim to train parameters of a pair of complementary classifiers. For example, corresponding parameters of each of the pair of hyperplanes 302A and 302B can be trained for estimating a data density corresponding to a normal distribution of a sequence of poses. The individual parameters of the pair of hyperplanes 302A and 302B can be denoted by (w1, b1) and (w2, b2), and the normal distribution of the sequence of poses 304 can be denoted by a data point x i .

[0055] In some example embodiments, the parameters (w1, b1) and (w2, b2) can be trained by solving the following equations (1), (2), (3), and (4).

[0056]

[0057] such that

[0058]

[0059] Equation (2) constrains a data point x i of the normal distribution of the sequence of poses 304 such that the data point x i belongs to the positive half-space of (w1, b1) (e.g., the hyperplane 302A). Likewise, equation (3) constrains a data point x i to the negative half-space of (w2, b2) (e.g., the hyperplane 302B). The notation is used for slack adjustment, and η > 0 specifies a classification margin for each of the pair of hyperplanes 302A and 302B.

[0060] Further, the parameters (w1, b1) and (w2, b2) can be constrained by equation (4) aiming to minimize a distance between the two hyperplanes 302A and 302B by β. For example, the distance between the two hyperplanes 302A and 302B corresponds to the Euclidean distance, which can be denoted by

[0061]

[0062] Here, w i denotes a weight that can not be constrained for one-class classification (e.g., data classification based on the discriminative one-class classifier 206). This can lead to overfitting of the data point x. To this end, in some embodiments, the weight w i can be regularized to a unit norm or unit vector, i.e., ||w1||2 = ||w2||2 = 1. In addition, the weights w1 and w2 can belong to a unit hyper-sphere U d-1 , which can be a sub-manifold of the Euclidean manifold .

[0063] Using Euclidean manifolds For a constrained submanifold, the optimization of equation (1) can be mathematically represented as follows:

[0064]

[0065] Using the unit norm constraint, dist2 simplifies to And α(b1, b2) = (b1 - b2) 2 -b1-b2. Notation [] + denoted as hinge loss, and v represents the penalty factor for the unity norm constraint.

[0066] Similarly, for a pair of orthogonal frames 306A and 306B, the corresponding subspace frames (W1 and W2) of orthogonal frames 306A and 306B can be trained to estimate data points x. i The density. For example, It is a subspace frame belonging to the set of all orthogonal k-frames, such as the Steifel manifold, by This indicates that there are K d-dimensional subspaces. The subspace frameworks W1 and W2 can be d×K dimensional matrices. Each matrix can include K columns, where each column of the matrix is ​​orthogonal to the remaining columns of the other matrices; that is, Where I K It is the identity matrix. Regarding W... i The orthogonality assumption ensures that each of the subspace frames W1 and W2 captures a different discriminative direction, leading to better regularization of the data points. Furthermore, the orthogonality of the subspace frames of the two standard orthogonal frames 306A and 306B also improves the representation of the data point distribution. P1 can be directly expanded to obtain:

[0067]

[0068] dist w It is the distance between standard orthogonal frames 306A and 306B. It is the bias vector. In equations (8) and (9), the notation W T x i +b is a K-dimensional vector. Therefore, equation (8) indicates The minimum value should be greater than the classification boundary (η) of each of the standard orthogonal frames 306A and 306B, as indicated by equation (9). The maximum value should be less than -η. For simplicity,

[0069] This is used to rewrite equation (6)P2 as:

[0070]

[0071] Due to the term The formulation of P'2enforces the coupling between the subspace frame and W2

[0072]

[0073] where Π is a K x K permutation matrix, belonging to the set of standard orthogonal matrices P k , defined as where denotes the space of binary numbers, and 1 is a vector of ones. For simplicity, assume that Π is a doubly stochastic matrix The set of all K x K positive definite matrices. The set of permutation matrices forms an angle of δ K Based on this assumption, equation (10) is mathematically rewritten as equation (12).

[0074]

[0075] In equation (12), the optimization over the distance between the standard orthogonal frames 304A and 304B to seek alignment between the subspace frames W1 and W2 can be computationally challenging. To this end, some embodiments use a measure that minimizes the Euclidean distance of each data point x i from both subspace frames W1 and W2. The minimization of the Euclidean distance of each input data point ensures that the distance between the subspace frames W1 and W2 bounds the entire data points x i To this end, the notation is mathematically rewritten as follows.

[0076]

[0077] In some embodiments, the sum of the lengths of each data point x is minimized after embedding onto the respective subspace frames (e.g., the standard orthogonal frames 306A and 306B). This enables the standard orthogonal frames 306A and 306B to be closer to each other to optimally bound the data points.

[0078] Further, using equation (13), the discriminative one-class classifier 206 can be mathematically represented as follows:

[0079]

[0080] Some embodiments are based on the recognition that the orthogonality constraint on the subspace frames W1 and W2 imposes a non-convex optimization problem on equation (14). To this end, some embodiments use a Riemannian conjugate gradient optimization technique to solve the non-convex optimization problem.

[0081] For illustration, assume F(W) is a function, where W represents a curved Riemannian manifold, the Riemannian gradient is mathematically expressed as follows:

[0082]

[0083] where, is the Euclidean gradient of F, which is mathematically expressed as follows.

[0084]

[0085]

[0086] where represents the indices of the straight hyperplanes of the corresponding subspace frames W1and W2; for equation (15), for equation (16), the variable is a d x K matrix, all of whose entries are zero except for the entries in the x i th column, which are set to x

[0087] In some example implementations, a pair of hyperspheres 308A and 308B can be trained for estimating data densities corresponding to a normal distribution of a pose sequence based on solving equations (6), (7), (8), (9), and (10). Mathematically, a pair of hyperspheres 308A and 308B can be expressed as follows:

[0088]

[0089] such that

[0090]

[0091] utilizing the fact that W T W = I K and using non-negative dual variables Equation (18) is mathematically expressed as follows.

[0092]

[0093] After simplifying equation (18), a pair of hyperspheres 308A and 308B is mathematically expressed as follows.

[0094]

[0095] such that (Y - Z)1 n = 0 equation (19)

[0096] ​

[0097] wherein is a linear kernel. The linear kernel can be replaced by any other positive definite kernel via the kernel trick (i.e., embedding the data points into a kernel).

[0098] Further, the approximation of a pair of hyperspheres 308A and 308B is mathematically represented as follows.

[0099]

[0100] wherein the generalized Stiefel manifold is defined as

[0101]

[0102] The mathematical relationship between the primal and dual variables is shown as follows.

[0103]

[0104]

[0105]

[0106]

[0107] wherein rowmax corresponds to the maximum value along the respective matrix's row and rowmin corresponds to the minimum value.

[0108] Accordingly, at step 410, the trained discriminative one-class classifier 206 provides as output a distribution model of the normalized pose sequence 304.

[0109] The trained discriminative one-class classifier 206 is used by the anomaly detector 106 for anomaly detection of human poses, which is described next with reference to Figure 5 .

[0110] Figure 5 A pipeline 500 depicting detection of anomalies in human poses by the anomaly detector 106 based on the trained discriminative one-class classifier 206, in accordance with some example embodiments of the present disclosure, is shown. The pipeline 500 is executed by the processor 208 of the anomaly detector 106, and the pipeline 500 begins at step 502.

[0111] At step 502, a sequence of poses (e.g., the sequence of poses 102) is provided to the anomaly detector 106. In example embodiments, the sequence of poses 102 can include normal poses of a person while performing an activity as well as abnormal poses. The sequence of poses 102 including normal poses and abnormal poses can be obtained from real-time video or pre-recorded video captured by an image capture device (e.g., a camera, a video recorder, a depth sensor camera, etc.).

[0112] At step 504, the pose sequence 102 is accepted as input data via the input interface 202 of the anomaly detector 106. The input data indicates a distribution of the pose sequence 102.

[0113] At step 506, the input data is embedded into a kernel space (e.g., Reproducing Kernel Hilbert Space (RKHS)). The input data is embedded into the RKHS for learning a pair of complementary classifiers. For example, a pair of complementary classifiers can be better learned to constrain normal data from different directions in a high-dimensional space such as the RKHS rather than a low-dimensional space. In some embodiments, the embedding of the input data into the RHKS can be pre-trained using a neural network. For example, a data point y of the input data can be embedded into the RKHS based on a trained data point (e.g., data point x) or a predefined kernel. If k(x,.) is a kernel mapping of a data point x and if a linear hyperplane in the kernel space has a form ∑x i w i k(x i ,.) (where w i is a weight assigned to each data point x i ), the kernel embedding can be pre-trained to embed the data point y i into the kernel space ∑x i w i k(x i , y i ).

[0114] At step 508, embedded data is obtained based on the kernel embedding of the input data into the RKHS.

[0115] At step 510, the embedded data is classified using the trained discriminant one-class classifier 206. In an example embodiment, the embedded data is classified based on learned parameters of a pair of complementary classifiers. The pair of complementary classifiers can bound data points corresponding to normal poses into in-class data from data points corresponding to abnormal poses. The data points corresponding to abnormal poses can be represented as out-of-class data.

[0116] In some example embodiments, a data point yi can be classified as an in-class data point if the following conditions are satisfied:

[0117] min(W1(x) + b1) > η ∧ max(W2(x) + b2) < -η,

[0118] where variables W and b are learned parameters of a pair of complementary classifiers of the discriminant one-class classifier 206.

[0119] At step 510, the classified embedded data is rendered as a classification result via the output interface 210. A representation of the classification result is shown in Figure 6 .

[0120] Figure 6 A graphical representation 600 depicting classification results corresponding to anomalies detected in the sequence of poses 102 is shown, in accordance with some example embodiments. In the graphical representation 600, data points 602 within the region bounded by a pair of standard orthogonal frames 606A and 606B represent in-class data (i.e., normal poses in the sequence of poses 102), and data points 604 outside the region bounded by the pair of standard orthogonal frames 606A and 606B represent out-of-class data (i.e., abnormal poses in the sequence of poses 102).

[0121] In some embodiments, anomaly detection in a sequence of poses (e.g., the sequence of poses 102) using the anomaly detector 106 can be implemented in a vehicle driver assistance system, which is described next with reference to Figure 7

[0122] Figure 7 An environmental representation 700 depicting anomaly detection in a vehicle driver assistance system 702 using the anomaly detector 106, in accordance with some example embodiments, is shown. In example embodiments, a vehicle 704 can be assisted by the vehicle driver assistance system 702. In some example embodiments, the vehicle driver assistance system 702 can be remotely connected to the vehicle 704 via a network 712. In some alternative embodiments, the vehicle driver assistance system 702 can be on-board the vehicle 704. Further, the vehicle driver assistance system 702 can be connected to the anomaly detector 106 via the network 712. In some alternative example embodiments, the anomaly detector 106 can be embodied within the vehicle driver assistance system 702.

[0123] The vehicle 704 can include an autonomous vehicle, a semi-autonomous vehicle, or a manual vehicle. In an illustrative example scenario, the vehicle 704 can be occupied by one or more passengers, such as a passenger 708A, a passenger 708B, a passenger 710A, and a passenger 710B, as Figure 7 shown. The vehicle 704 can be equipped with a camera 706 (e.g., a dashcam) to capture a sequence of image frames having poses of the passengers 708A, 708B, 710A, and 710B. The passengers 708A and 708B can be seated in a front seat of the vehicle 704, and the passengers 710A and 710B can be seated in a back seat of the vehicle 704. The camera 706 can be coupled to the vehicle driver assistance system 702 for providing the captured sequence of image frames having poses of the passengers 708A, 708B, 710A, and 710B.

[0124] ​In some cases, it can not be feasible to capture the poses of all passengers 708A, 708B, 710A, and 710B through the camera 706. For example, the poses of the backseat passengers 710A and 710B captured through the camera 706 can not be accurate due to factors such as occlusion, the environment of the backseat, etc. In such cases, the poses of the passengers 710A and 710B can be filtered or discarded, and the poses of the passengers 708A and 708B can be considered only for anomaly detection.

[0125] In some other cases, the vehicle 704 can be equipped with multiple cameras in the vehicle 704 to capture the poses of all passengers 708A, 708B, 710A, and 710B in the vehicle 704. For example, another camera (not shown in FIG. 7) can be equipped in the vehicle 704 for capturing the set of image frames of the poses of the passengers 710A and 710B having the backseat of the vehicle. In this way, the camera 706 can capture the poses of the passengers 708A and 708B having the frontseat of the vehicle, and the other camera can capture the poses of the passengers 710A and 710B having the backseat of the vehicle. Figure 7

[0126] The camera 706 can send the captured sequence of image frames of the poses of the passengers 708A, 708B, 710A, and 710B to the vehicle driver assistance system 702, which can further send it to the anomaly detector 106. The poses of the passengers 708A, 708B, 710A, and 710B can include sitting poses, driving poses, dancing poses, etc.

[0127] The anomaly detector 106 detects anomalous poses in the poses of the passengers 708A, 708B, 710A, and 710B, which is next described in Figure 8

[0128] Figure 8 A block diagram 800 depicting detection of anomalous poses in the sequence of image frames 802 by the anomaly detector 106, in accordance with some example embodiments, is shown. In some example embodiments, the sequence of image frames 802 corresponds to the sequence of image frames of the poses of the passengers 708A, 708B, 710A, and 710B captured through the camera 706. The sequence of image frames 802 is provided as input to the anomaly detector 106.

[0129] At step 804, the anomaly detector 106 detects the poses in each of the sequence of image frames 802 based on one of the pose estimation techniques. Some of the pose estimation techniques can include OpenPose, DeepCut pose estimation, Region-based Multi-Person Pose Estimation (RMPE), Mask RCNN, etc.

[0130] ​​In step 806, the anomaly detector 106 filters a set of image frames with pose sequences (e.g., the poses of passengers 708A, 708B, 710A, and 710B) from the image frame sequence 802 based on the detected poses. In the filtering step, the poses of one or more passengers (e.g., passengers 710A and 710B) affected by oscillations, the rear-seat environment, or other factors are filtered.

[0131] After filtering the image set, in step 806, the anomaly detector 106 performs preprocessing on the image frame set. In preprocessing step 806, the anomaly detector 106 matches the pose of an image frame in the image frame set with another pose in the next image frame in the image frame set that is spatially close to the pose of the image frame in the image frame set. In some example implementations, the pose in the current image frame may be matched with the pose in the next image frame based on a time-sliding window. In some implementations, the time-sliding window may be pre-trained to generate a fixed-length pose sequence. The time-sliding window may be pre-trained based on a predefined dataset comprising image frames learned from various video clips. For example, the image frame set may have a large number of image frames (e.g., 1000 image frames), where only some image frames may include the actual poses of one or more passengers (e.g., passengers 708A and 708B). These image frames with actual poses may be generated as a fixed length (e.g., 500 image frames) based on the trained time-sliding window. In some example implementations, the time-sliding window may include a pose bounding box for matching the pose from one image frame with the pose in the next image frame. After determining the matching poses, a unique pose sequence for each passenger in vehicle 704 (e.g., passenger 708A and passenger 708B) is generated.

[0132] In step 808, the anomaly detector 106 performs normalization on the generated pose sequence. To this end, the anomaly detector 106 prunes joints from the poses detected in each pose sequence. In some embodiments, joints are pruned based on the detection score of the corresponding joint in the detected pose being below a predefined threshold. For example, passenger 708B may be in a seated position with knees on the seat. Joints in this seated position may have low detection scores (e.g., 4), which may be below a predefined threshold (e.g., 10). Pose with low joint detection scores is pruned. In some example embodiments, each joint may be associated with a visibility marker. The visibility marker can indicate whether a joint is occluded. Therefore, the anomaly detector 106 may also prune joints based on the visibility marker of the corresponding joint. In some example embodiments, the anomaly detector 106 may assign more weight to the joint's visibility marker than to the joint's detection score. Therefore, the anomaly detector 106 can avoid pruning visible joints with low detection scores.

[0133] After pruning the joints with lower detection scores and / or the joints with visibility flags indicating that the joints are occluded, the joints in the respective sequence of poses with detection scores higher than a predefined threshold and / or the joints with visibility flags indicating that the joints are not occluded are normalized. In example implementations, the joints in the respective sequence of poses can be normalized between [0, 1] by dividing the positions of the respective joints by the size of the image frames in the sequence of image frames. The division of the positions is followed by subtracting the neck joint from the respective joint positions of the respective poses in the sequence of poses. The neck joint represents the origin of the respective poses.

[0134] At step 810, the anomaly detector 106 generates the normalized sequence of poses as an output.

[0135] In some implementations, the anomaly detector 106 generates a vector representation for each pose in the normalized sequence of poses. The respective vector representations of each pose in the normalized sequence of poses are embedded into a sequence representation of a predefined dimension using the discriminative one-class classifier 206. For example, the vector representation can represent each pose in a total of 2n-dimensional (x, y) position coordinates, where n is the number of all joints in the pose. In case of occluded joints, the corresponding position coordinates of the occluded joints can be assigned as 0. In some example implementations, the sequence representation can correspond to a normalized histogram. The normalized sequence of poses and the normalized histogram are next described with reference to Figure 9

[0136] Figure 9 FIG. 9 shows a diagram 900 depicting a normalized sequence of poses 902 and a normalized histogram 904, according to some example implementations of the present disclosure. The normalized sequence of poses 902 is mapped into the normalized histogram 904. The normalized histogram 904 can include one or more bins that can be obtained based on k-means clustering of pre-trained human poses. In some implementations, the normalized histogram 904 is mapped into a pose bag 906 based on a predefined notation of the poses associated with the trained human poses.

[0137] Figure 10 FIG. 10 shows a method flow 1000 for detecting anomalies in a sequence of poses of a person performing an activity, according to some example implementations of the present disclosure. The method 1000 is performed by the anomaly detector 106. At operation 1002, input data indicative of a distribution of a sequence of poses (e.g., the normalized sequence of poses 802) is accepted. In some example implementations, the distribution of the sequence of poses 802 corresponds to a set of images of a person performing an activity. In some other example implementations, the input data can correspond to a sequence of image frames including one or more images of one or more passengers (e.g., the passengers 708A, 708B, 710A, and 710B) in a vehicle (e.g., the vehicle 704).

[0138] ​At operation 1004, the input data is embedded into elements of the RKHS. The RKHS provides a linear space that allows for a non-linear projection of the input data mapped into the linear space.

[0139] At operation 1006, the embedded data is classified using the discriminative one-class classifier 206. The discriminative one-class classifier includes a pair of complementary classifiers that bound a normal distribution of the gesture sequence in the RKHS. Examples of the pair of complementary classifiers include a pair of hyperplanes 302A and 302B, a pair of orthonormal frames 304A and 304B, a pair of hyperspheres 308A and 308B, etc. In some implementations, the discriminative one-class classifier 206 is trained based on a classification model. The classification model can correspond to at least a bag-of-words model, a convolutional neural network (CNN) model, or a deep learning model. In some example implementations, the discriminative one-class classifier 206 is trained to estimate a data density of the normal distribution of the gesture sequence based on a min-max optimization technique. The min-max optimization technique is used to minimize a distance between individual classifiers of the pair of complementary classifiers and to maximize a margin of individual classifiers of the pair of complementary classifiers from normal data for a maximum encapsulation of the normal distribution of the gesture sequence. The min-max optimization technique corresponds to a Riemannian conjugate gradient algorithm.

[0140] At operation 1008, the classification result is rendered via the output interface 210 of the anomaly detector 106. The classification result can correspond to an anomaly detected in the gesture sequence. The anomaly detected in the gesture sequence can be notified to the person performing the activity.

[0141] Figure 11A A use case 1100 implementation of the anomaly detector 106 according to some example implementations of the present disclosure is shown. The use case 1100 corresponds to a vehicle driver assistance system, such as the vehicle driver assistance system 702 of Figure 7 The anomaly detector 106 can detect abnormal gestures of one or more passengers in the vehicle, such as the passenger 708A and the passenger 708B (also referred to as the passengers 708A and 708B) in the vehicle 704.

[0142] In an illustrative example scenario, the passenger 708A driving the vehicle 704 can turn away from looking straight ahead at the road. These turned-away gestures can be captured by the camera 706. The camera 706 can provide these gestures to the anomaly detector 106 via the vehicle driver assistance system 702. The anomaly detector 106 can detect these gestures as abnormal gestures and send an alert notification 1102 to the passenger 708A based on the detected abnormal gestures. The alert notification 1102 can include a combination of audio, visual, or audio-visual notifications, such as “Anomaly detected!” Additionally or alternatively, the alert notification 1102 can be followed by a reminder 1104 of the message “Stay alert, safe driving.”

[0143] In some cases, both passengers 708A and 708B can move and exhibit a potentially abnormal posture. In these cases, the abnormality detector 106 can detect the abnormal posture of each of the passengers 708A and 708B and generate an alert notification 1102 for each of the passengers 708A and 708B. In some example implementations, the abnormality detector 106 can identify the posture of the passengers 708A and 708B based on human action recognition techniques, human activity recognition techniques, and the like.

[0144] In a similar manner, abnormality detection of human postures can be implemented in a security monitoring system, which is described in Figure 11B .

[0145] Figure 11B A use case 1108 implementation of the abnormality detector 106 according to some other example implementations of the present disclosure is shown. The use case 1108 corresponds to a security monitoring system 1120 at a location 1110. The location 1110 can include an indoor area (e.g., a shopping mall, a bank, a hospital care unit, and the like), an outdoor area (e.g., a park, a roadside, and the like), and the like. The location 1110 can be equipped with one or more cameras, such as cameras 1112A and 1112B.

[0146] In an illustrative example scenario, the location 1110 corresponds to a commercial area such as a shopping mall, equipped with cameras 1112A and 1112B installed at different points of the shopping mall. For example, the camera 1112A can be installed at an entrance of the location 1110, and the camera 1112B can be installed at a different point in the location 1110, as shown in Figure 11B Each of the cameras (i.e., the camera 1112A and the camera 1112B) captures events occurring in the location 1110. The captured events can include people performing different activities. These captured events are provided to the security monitoring system 1120. The security monitoring system 1120 is equipped with the abnormality detector 106. The abnormality detector 106 processes the captured events and detects abnormal events based on detecting abnormal postures in different activities of the people.

[0147] For example, a person 1114 falls down while walking towards the location 1110. The fall down event of the person 1114 is captured by the camera 1112A. The captured fall down event is streamed to the security monitoring system 1120. The security monitoring system 1120 streams the fall down event to the abnormality detector 106. The abnormality detector 106 detects the fall down posture as an abnormal posture from a normal walking posture. The abnormality detector 106 notifies an operator of the security monitoring system 1120 of an alert signal based on the detected abnormal posture. The operator can include a security force notified to assist the person 1114.

[0148] In some cases, unwanted events such as fighting, theft, etc. can also occur in location 1110. These unwanted events can also include human activities with postures that can be discriminated as abnormal. For example, in front of the field of view of camera 1112B, person 1116 can attack another person 1118. Person 1116 can initially be in a running posture, followed by a kicking posture. Person 1118 can initially be in a normal walking posture, and suddenly start a running posture due to the attack by person 1116. Camera 1112B can capture the video stream of these postures and send it to security monitoring system 1120. Security monitoring system 1120 can send the video stream for processing by anomaly detector 106. Anomaly detector 106 detects the kicking posture of person 1116 as an abnormal posture, and the running posture of person 1118 as an abnormal posture. Anomaly detector 106 can notify security monitoring system 1120 based on the detected abnormal postures. Security monitoring system 1120 alerts an operator based on the notification. In this way, the operator can handle and control these unwanted events in a feasible and efficient way without having to manually monitor security monitoring system 1120.

[0149] Alternatively or additionally, anomaly detector 106 can be implemented in different applications. For example, anomaly detector 106 can support processing of complex and non-linear data corresponding to different modalities (e.g., audio-based input data, text-based input data, etc.). Anomaly detector 106 can detect abnormal audio in audio-based input data, which can be applied to detecting malfunctioning equipment, music coaching applications, etc. In a similar manner, anomaly detector 106 can detect abnormal text in text-based input data, which can be applied to fraud detection in text documents, fraud detection in file transactions, etc.

[0150] A general block diagram of anomaly detector 106 is shown and described in Figure 12 .

[0151] Figure 12 A general block diagram of anomaly detector 1200 according to some example embodiments of the present disclosure is shown. Anomaly detector 1200 corresponds to anomaly detector 106 of Figure 1 . Anomaly detector 1200 includes a processor 1204 configured to execute stored instructions, and a memory 1206 storing instructions executable by processor 1204. In some embodiments, memory 1206 is also configured to store a discriminative one-class classifier 1208. Discriminative one-class classifier 1208 corresponds to discriminative one-class classifier 206. Discriminative one-class classifier 1208 includes a pair of complementary classifiers.

[0152] In some example implementations, the processor 1204 accesses the discriminative one-class classifier 1208 for performing anomaly detection in a sequence of human poses of a performing activity. The processor 1204 corresponds to the processor 208. The processor 1204 can be a single core processor, a multi-core processor, a computing cluster, or any number of other configurations. The memory 1206 can include random access memory (RAM), read only memory (ROM), flash memory, or any other suitable memory systems. The processor 1204 is connected through a bus 1214 to the input interface 1202. These instructions implement the method 1000 for detecting anomalies in a sequence of human poses of a performing activity, e.g., the anomaly detection described in the use case 1100 of Figure 11A the use case 1108 of Figure 11B the use case 1108 of

[0153] In some implementations, the anomaly detector 1200 can have different types and combinations of input interfaces to receive the input data 1210. In one implementation, the input interface 1202 can include an audio-video receiver (AVR), a keyboard, and / or a pointing device, such as a mouse, trackball, touchpad, joystick, pointing stick, stylus, touch screen, etc.

[0154] Additionally or alternatively, the network interface controller 1212 can be adapted to connect the anomaly detector 1200 to a network 1222 through the bus 1214. Through the network 1222, the input data 1210 can be downloaded and stored within the memory 1206 for storage and / or further processing.

[0155] Additionally or alternatively, the anomaly detector 1200 can include a storage device 1220 for storing the trained parameters of the pair of complementary classifiers of the discriminative one-class classifier 1208, the annotated pose bags in the input data 1210 for anomaly detection, etc.

[0156] In addition to the input interface 1202, the anomaly detector 1200 can include one or more output interfaces to output the classification results rendered from the anomaly detection. For example, the anomaly detector 1200 can be linked through the bus 1214 to an output interface 1216, which is adapted to connect the anomaly detector 1200 to an output device 1218. The output device 1218 can include a computer monitor, a projector, a display device, a screen, a mobile device, an audio device, etc.

[0157] In this way, the anomaly detector 1200 uses a one-class classifier (e.g., the discriminative one-class classifier 1208) to detect anomalous poses in a sequence of human poses. The discriminative one-class classifier 1208 uses its pair of complementary classifiers to define the normal distribution of a sequence of poses in an accurate manner, regardless of the shape of the normal distribution of the sequence of poses. Further, the pair of complementary classifiers is learned in a reproducing Hilbert kernel space that ensures that the boundary of the pair of complementary classifiers is linear, which improves the computation of anomaly detection in a sequence of human poses. Further, anomaly detection can be performed without the availability of training data corresponding to anomalous human poses. This reduces the computational complexity of anomaly detection of human activities.

[0158] The above-described embodiments of the present disclosure can be implemented in any of numerous ways. For example, the embodiments can be implemented using hardware, software or a combination thereof. When implemented in software, the software code can be executed on any suitable processor or collection of processors, whether provided in a single computer or distributed among multiple computers. Such processors can be implemented as integrated circuits, with one or more processors in an integrated circuit component. However, a processor can be implemented using circuitry in any suitable format.

[0159] Additionally, the various methods or processes outlined herein can be coded as software that is executable on one or more processors that employ any one of a variety of operating systems or platforms. Additionally, such software can be written using any of a number of suitable programming languages and / or programming or scripting tools, and also can be compiled as executable machine language code or intermediate code that is executed on a framework or virtual machine. Typically, the functionality of the program modules can be combined or distributed as desired in various embodiments.

[0160] In addition, embodiments of the present disclosure can be embodied as a method, of which an example has been provided. The acts performed as part of the method can be ordered in any suitable way. Accordingly, embodiments can be constructed in which acts are performed in an order different than illustrated, which can include performing some acts simultaneously, even though shown as serial acts in illustrative embodiments. The appended claims are intended to cover and embrace all such alterations and modifications as can come within the true spirit and scope of this disclosure.

Claims

1. An anomaly detector for detecting an anomaly in a sequence of poses of a person performing an activity, the anomaly detector comprising: an input interface configured to accept input data indicative of a distribution of the sequence of poses; a memory configured to store a discriminative one-class classifier of a pair of complementary classifiers having a normal distribution of sequences of poses defined in a Reproducing Kernel Hilbert Space (RKHS); a processor configured to: embed the input data into elements of the RKHS; and classify the embedded data using the discriminative one-class classifier; and an output interface configured to render a classification result, wherein the sequence of poses corresponds to a set of image frames of the person performing the activity, wherein the input data indicative of the distribution of the sequence of poses corresponds to a set of images of the person performing the activity, wherein the normal distribution of sequences of poses corresponds to a distribution of normal sequences of poses of the person while performing the activity, wherein the processor is further configured to detect an anomaly in the sequence of poses of the person performing the activity based on the classified embedded data, wherein the anomaly is classified from normal data corresponding to a normal distribution of sequences of poses based on the discriminative one-class classifier, wherein the discriminative one-class classifier detects outliers from the normal distribution of sequences of poses defined as anomalies of human poses, wherein the discriminative one-class classifier is trained based on a classification model, wherein the classification model corresponds to at least a bag-of-words model or a deep learning model, wherein the discriminative one-class classifier is trained to estimate a data density of the normal distribution of sequences of poses based on a min-max optimization technique, and wherein the min-max optimization technique is used to: minimize a distance between individual classifiers of the pair of complementary classifiers, and maximize a margin of individual classifiers of the pair of complementary classifiers for a maximum encapsulation of the normal distribution of sequences of poses.

2. The anomaly detector of claim 1, wherein, the processor is further configured to output a notification to the person performing the activity based on detecting the anomaly.

3. The anomaly detector of claim 1, wherein, the min-max optimization technique corresponds to a Riemannian conjugate gradient algorithm.

4. The anomaly detector of claim 1, wherein, the input data corresponds to a sequence of image frames comprising one or more images of one or more passengers in a vehicle.

5. The anomaly detector of claim 4, wherein, the one or more images of the one or more passengers are captured by one or more image capturing devices installed in the vehicle.

6. The anomaly detector of claim 1, wherein, the processor is configured to: detect poses in individual sequences of image frames based on one of pose estimation techniques, the sequences of image frames corresponding to the input data and comprising one or more images of one or more passengers in a vehicle; filter a set of image frames having the sequence of poses from the sequences of image frames based on the detected poses; match a pose in an image frame of the set of image frames with another pose in a next image frame of the set of image frames, wherein the pose in the image frame is spatially close to the other pose in the next image frame; and generating the sequence of poses from the filtered set of image frames based on matching poses in the set of image frames; wherein the processor is further configured to: prune joints from the detected poses in each of the sequence of poses based on a detection score of a corresponding joint in the detected poses being below a predefined threshold; normalize remaining joints in the detected poses in each of the sequence of poses, wherein the remaining joints correspond to joints with a detection score above the predefined threshold; and generate a vector representation corresponding to position coordinates of each pose in the normalized sequence of poses; wherein the processor is further configured to: embed each vector representation of each pose in the normalized sequence of poses into a single sequence representation of a predefined dimension using the discriminative one-class classifier as a normalized histogram; and wherein the processor is further configured to map the normalized histogram into a pose bag based on a predefined notation of poses associated with a trained sequence of poses of a human activity, and wherein the normalized histogram comprises one or more bars obtained based on clustering of the trained sequence of poses of the human activity.

7. A method for detecting anomalies in a sequence of poses of a person performing an activity, the method comprising the steps of: accept input data indicative of a distribution of the sequence of poses; embedding the input data into a Reproducing Kernel Hilbert Space (RKHS); classifying the embedded data using a discriminative one-class classifier, wherein the discriminative one-class classifier comprises a pair of complementary classifiers defining a normal distribution of sequences of poses in the RKHS; and rendering a classification result, wherein the sequence of poses corresponds to a set of image frames of the person performing the activity, wherein the input data indicative of the distribution of the sequence of poses corresponds to a set of images of the person performing the activity, wherein the normal distribution of sequences of poses corresponds to a distribution of normal sequences of poses of the person while performing the activity, wherein the method further comprises detecting an anomaly in the sequence of poses of the person performing the activity based on the classified embedded data, wherein the anomaly is classified from normal data corresponding to a normal distribution of sequences of poses based on the discriminative one-class classifier, wherein the discriminative one-class classifier detects outliers from the defined normal distribution of sequences of poses as anomalies of human poses, wherein the method further comprises: training the discriminative one-class classifier based on a classification model, wherein the classification model corresponds to at least a bag-of-words model or a deep learning model; and estimating a data density of the normal distribution of sequences of poses based on a min-max optimization technique, wherein the min-max optimization technique is used to: minimize a distance between each of the pair of complementary classifiers, and maximize a margin of each of the pair of complementary classifiers.

8. The method of claim 7, further comprising the step of: outputting a notification to the person performing the activity based on detecting the anomaly.

9. The method of claim 7, wherein The min-max optimization technique corresponds to the Riemannian Conjugate Gradient algorithm.

10. The method of claim 9, further comprising the steps of: capturing, by one or more image capturing devices installed in a vehicle, a sequence of image frames comprising one or more images of one or more passengers in the vehicle; wherein the method further comprises the steps of: detecting poses in each of the sequence of image frames based on one of the pose estimation techniques; filtering a set of image frames having the sequence of poses from the sequence of image frames based on the detected poses; and matching a pose in an image frame in the set of image frames with another pose in a next image frame in the set of image frames; and generating the sequence of poses from the filtered set of image frames based on the matched poses in the set of image frames; wherein the method further comprises the steps of: pruning joints from the detected poses in each of the sequence of poses based on a detection score of a corresponding joint in the detected poses being below a predefined threshold; normalizing remaining joints in the detected poses in each of the sequence of poses, wherein the remaining joints correspond to joints having a detection score above the predefined threshold; and generating a vector representation corresponding to position coordinates of each pose in the normalized sequence of poses; embedding each vector representation of each pose in the normalized sequence of poses into a single sequence representation of a predefined dimension as a normalized histogram using the discriminative one-class classifier; and mapping the normalized histogram into a pose bag based on a predefined notation of poses associated with a trained sequence of poses of a human activity, wherein the normalized histogram comprises one or more bars obtained based on clustering of the trained sequence of poses of the human activity.

Images