Quantitative Transition Change Detection for Activity Recognition

Abstract

A system for identifying human activities in a video stream, comprising a classifier for classifying image frames of the video stream into one or more categories and generating a category probability vector for the image frame based on the classification. The system further includes a data filtering and binarization module for performing filtering and binarization processing on each probability value of the category probability vector based on a predefined probability threshold. The system further includes a compressed word synthesis module for determining one or more transitions of one or more categories of one or more consecutive image frames of the video stream based on the corresponding binarized probability vectors and generating a compressed word sequence based on the determined one or more transitions. The system further includes a sequence-dependent classifier for extracting one or more user actions by analyzing the compressed word sequence and identifying human activities therefrom.

Description

Technical Field

[0001] The present disclosure mainly relates to artificial intelligence, and particularly to identifying human activities from video streams and symbolic processing. Background Art

[0002] With the progress of technology, human activity recognition has become extremely important. Human activity recognition is helpful for various applications, such as monitoring the checkout process in a retail store involving a self-checkout (SCO) system. Such a system allows buyers to complete the purchase process by themselves. Another example of an application of human activity recognition is to assist video surveillance by detecting unfair activities (such as theft) committed by shoplifters, thereby alerting the personnel employed in the store to prevent theft. In addition, human activity recognition is also used in intelligent driving assistance systems, assisted living systems for people in need, video games, and physical therapy, etc. Additionally, human activity recognition is actively used in fields such as sports, military, medical, and robotics.

[0003] Human activities represent most of the components for process modeling. However, since human behavior is unpredictable, it is a difficult task to identify such human activities in diverse environments. Human activities can generally be decomposed into a set of basic actions involving different human body parts (such as hands, feet, and faces, etc.). In addition, the set of basic actions associated with human activities spans multiple time intervals. The recognition task of such activities faces the problem of summarizing the entire action sequence over variable time intervals.

[0004] Since each person has a different body structure, different body types, different skin colors, etc., traditional human activity recognition technologies are very inefficient in identifying human activities. Also, the time range of human activities can vary significantly over time depending on the subject and possibly other environmental conditions. In addition, not all basic body part movements are associated with the purpose of the activity being considered. Therefore, the activity recognition process faces two main problems related to the temporal variations in the actions of the human body parts involved in the activity and the physical trajectory variations.

[0005] Therefore, in view of the above discussion, it is necessary to overcome the above-mentioned defects related to human activity recognition, and to provide a system and method aimed at reducing the impact of temporal variations and body part movements on activity recognition using recurrent neural networks. Summary of the Invention

[0006] The present disclosure aims to provide a system and a method for identifying human activities from video streams.

[0007] According to one aspect of the present disclosure, there is provided a system for identifying human activities from a video stream captured by an imaging device. The system includes a memory for storing one or more instructions, and a processor communicatively coupled to the memory. The system includes a classifier communicatively coupled to the imaging device and configured to classify image frames of the video stream into one or more categories from a predefined set of categories, wherein the image frames are classified based on user actions in the region of interest of the image frame, and to generate a category probability vector for the image frame based on the classification, wherein the category probability vector includes a set of classification probabilities of the image frame in each predefined category. The system further includes a data filtering and binarization module configured to perform filtering and binarization processing on each probability value of the category probability vector based on a predefined probability threshold. The system further includes a compressed word synthesis module configured to determine one or more transitions of one or more categories in one or more consecutive image frames of the video stream based on the corresponding binarized probability vector, and to generate a compressed word sequence based on the determined one or more transitions in the one or more consecutive image frames. The system further includes a sequence-dependent classifier configured to extract one or more user actions by analyzing the compressed word sequence and to identify human activities therefrom.

[0008] According to another aspect of the present disclosure, there is provided a method for identifying human activities from a video stream. The method includes classifying image frames of the video stream into one or more categories from a predefined set of categories by a classifier, wherein the image frames are classified based on user actions in the region of interest of the image frame. The method further includes generating a category probability vector for the image frame based on the classification, wherein the category probability vector includes a set of classification probabilities of the image frame in each predefined category. The method further includes binarizing each probability value of the category probability vector based on a predefined probability threshold. The method further includes determining one or more transitions of one or more categories in one or more consecutive image frames of the video stream based on the corresponding binarized probability vector. The method further includes generating a compressed word sequence based on the determined one or more transitions in the one or more consecutive image frames. The method further includes extracting one or more user actions by analyzing the compressed word sequence and identifying human activities therefrom.

[0009] According to another aspect of the present disclosure, there is provided a computer-programmable product for identifying human activities from a video stream, the computer-programmable product including a set of instructions. When executed by a processor, the set of instructions causes the processor to classify image frames of the video stream into one or more categories from a predefined set of categories, wherein the image frames are classified based on user actions in the region of interest of the image frame, generate a category probability vector for the image frames based on the classification, wherein the category probability vector includes a set of classification probabilities of the image frame in each predefined category, perform binarization processing on each probability value of the category probability vector based on a predefined probability threshold, determine one or more transitions of one or more categories in one or more consecutive image frames of the video stream based on the corresponding binarized probability vector, generate a compressed word sequence based on the determined one or more transitions in one or more consecutive image frames, and extract one or more user actions by analyzing the compressed word sequence, and identify human activities therefrom.

[0010] The present disclosure aims to provide a system for identifying human activities from a video stream. Such a system can effectively and reliably identify human activities from a video stream.

[0011] It should be understood that the features of the present disclosure can be easily combined in various combinations without departing from the scope of the present disclosure defined by the appended claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The above-described invention content and the following detailed description of illustrative embodiments can be better understood by reading in conjunction with the accompanying drawings. For the purpose of illustrating the present disclosure, exemplary configurations of the present disclosure are shown in the drawings. However, the present disclosure is not limited to the specific methods and means disclosed herein. In addition, those skilled in the art will understand that the drawings are not drawn to scale. Where possible, the same elements are denoted by the same reference numerals.

[0013] Embodiments of the present disclosure will now be described by way of example with reference to the following drawings, in which:

[0014] Figure 1 shows an environment in which different embodiments of the present disclosure can be implemented;

[0015] Figure 2 shows an activity recognition system for identifying one or more human actions and activities in a video stream captured by an Figure 1 imaging device according to an embodiment of the present disclosure; and

[0016] Figure 3 is a flowchart showing a method for identifying human activities from a video stream according to an embodiment of the present disclosure.

[0017] In the drawings, underlined numbers are used to represent the item in which the underlined number is located or the item adjacent to the underlined number. Numbers without underlines are associated with the item identified by the line connecting the non-underlined number to an item. When a number has no underline and is accompanied by a related arrow, the non-underlined number is used to identify the general item pointed to by the arrow. Detailed Description

[0018] The following detailed description shows embodiments of the present disclosure and ways in which they can be implemented. Although some modes for implementing the present disclosure are disclosed, those skilled in the art will recognize that other embodiments for implementing or practicing the present disclosure are also possible.

[0019] Figure 1 An environment 100 is shown in which different embodiments of the present disclosure can be implemented. The environment 100 includes an imaging device 101, an activity recognition system 102, and a computing device 103 that are communicatively coupled to each other via a communication network 104. The communication network 104 can be any suitable wired network, wireless network, combination of these networks, or any other conventional network without limiting the scope of the present disclosure. A few examples can include a local area network (LAN), a wireless LAN connection, an Internet connection, a point-to-point connection, or other network connections and combinations thereof.

[0020] The imaging device 101 is configured to capture a video stream. In one embodiment of the present disclosure, the imaging device 101 is configured to capture one or more images of a retail checkout process including a self-checkout system (SCO). Optionally, the imaging device 101 includes, but is not limited to, an Internet Protocol (IP) camera, a pan-tilt-zoom (PTZ) camera, a thermal imaging camera, or an infrared camera.

[0021] The activity recognition system 102 is configured to identify human actions and human activities in the video stream captured by the imaging device 101.

[0022] The activity recognition system 102 includes a central processing unit (CPU) 106, an operation panel 108, and a memory 110. The CPU 106 is a processor, a computer, a microcontroller, or other circuitry that controls the operation of different components, such as the operation panel 108 and the memory 110. The CPU 106 can execute software, firmware, and / or other instructions, such as software, firmware, and / or other instructions stored in a volatile or non-volatile memory, such as the memory 110, or otherwise provided to the CPU 106. The CPU 106 can be connected to the operation panel 108 and the memory 110 through a wired or wireless connection, such as one or more system buses, cables, or other interfaces. In one embodiment of the present disclosure, the CPU 106 can include custom graphics processing unit (GPU) server software for providing real-time object detection and prediction for all cameras on a local network.

[0023] The operation panel 108 can be a user interface of the image forming apparatus 100 and can be in the form of a physical keyboard or a touch screen. The operation panel 108 can receive inputs related to selected functions, preferences, and / or authentication from one or more users and can provide and / or receive inputs in a visual and / or auditory manner.

[0024] In addition to storing instructions and / or data for the CPU 106 to manage the operation of the image forming apparatus 100, the memory 110 can also include user information associated with one or more users of the image forming apparatus 100. For example, the user information can include authentication information (such as a username / password pair), user preferences, and other user-specific information. The CPU 106 can access this data to assist in providing control functions related to the operation of the operation panel 108 and the memory 110 (such as sending and / or receiving one or more control signals).

[0025] The imaging device 101 and the activity recognition system 102 can be controlled / operated by a computing device 103. Examples of the computing device 103 include a smart phone, a personal computer, and a laptop computer, etc. The computing device 103 enables a user / operator to view and save videos captured by the imaging device 101, and to access videos / images processed by the activity recognition system 102. The computing device 103 can execute a mobile application of the activity recognition system 102 to enable a user to access and process video streams captured by the imaging device 101.

[0026] In one embodiment, the camera 101, the activity recognition system 102, and the computing device 103 can be integrated in a single device, where the single device is a portable smart phone with a built-in camera and a display.

[0027] Figure 2An activity recognition system 102 for identifying one or more human actions and activities in a video stream captured by an imaging device 101 according to an embodiment of the present disclosure is shown.

[0028] The activity recognition system 102 includes a CPU 106, and the CPU 106 includes a classifier 202 that can be operated to analyze each frame of the video stream to determine at least one action region of interest, where the at least one region of interest includes at least one object. The action region of interest refers to a rectangular region in each frame of the video stream, where at least one object and one or more actions that occur can be seen in the region. In one example, the at least one object can be a person, an item (such as clothing, groceries, and a wallet, etc.), and the one or more actions can include a person taking out a wallet from a pocket, a person queuing, and a person swiping a credit card, etc. Each action can be used as a component for process model extraction, and a process can be represented as a chain of actions.

[0029] In an embodiment of the present disclosure, the classifier 202 can be an algorithm-based classifier, such as a convolutional neural network (CNN) that can be trained to classify image frames of a video of an SCO scanning region (scanning action region of interest) (such as classifying into categories such as hand, object in hand, object, body part, empty scanner, etc.). The classification criteria for image frames in each category are mentioned below:

[0030] Hand - The image frame shows a human hand.

[0031] Object in hand - The image frame shows an object in the user's hand.

[0032] Object - The image frame shows only the object.

[0033] Body part - The image frame shows a human body part.

[0034] Empty scanner - The image frame shows only an empty scanner.

[0035] The CNN 200 referred to herein is defined as a trained deep artificial neural network, mainly used for classifying at least one object in at least one region of interest. In particular, they are algorithms that can recognize faces, individuals, street signs, etc. The term "neural network" used herein may include a network of highly interconnected processing elements, where each processing element is optionally associated with local memory. In one example, the neural network can be a Kohonen network, a multi-layer perceptron, etc. In addition, the processing elements of the neural network can be "artificial neural units", "artificial neurons", "neural units", "neurons", and "nodes", etc. In addition, a neuron can receive data from an input neuron or one or more other neurons, process the data, and send the processed data to an output neuron or one or more other neurons. The neural network or one or more neurons can be generated in hardware, software, or a combination of hardware and software, and the neural network can then be trained. It should be understood that a convolutional neural network (CNN) consists of an input layer, multiple hidden layers, and an output layer. In addition, the multiple hidden layers of a convolutional neural network usually consist of convolutional layers, pooling layers, fully connected layers, and normalization layers. Optionally, the Visual Geometry Group 19 (VGG19) model is used as the convolutional neural network architecture. The VGG19 model is configured to classify at least one object in a frame of a video stream. It should be understood that the hidden layer includes multiple sets of convolutional layers.

[0036] In operation, the classifier 202 receives an image frame of a video stream regarding the SCO scan region (the scanned action region of interest), and classifies it into categories such as hand, object in hand, object, body part, empty scanner, etc. based on the content of the image frame. In one embodiment of the present disclosure, the classifier 202 statically analyzes each image frame and outputs a category probability vector P with one component for each category considered for each image frame v , where P v = {P Hand , P HandObject , P Object , P BodyPart , P EmptyScanner}}.

[0037] Where P Hand = the probability that the image frame is classified into the category "hand"

[0038] P HandObject = the probability that the image frame is classified into the category "object in hand"

[0039] P Object = the probability that the image frame is classified into the category "object"

[0040] P BodyPart= Probability that an image frame is classified into the category "body part"

[0041] P EmptyScanner = Probability that an image frame is classified into the category "empty scanner"

[0042] In one example, classifier 202 generates six probability vectors P for six consecutive image frames in five categories v1 to P v6 , in the format shown below.

[0043] P v1 = {0.0, 0.0, 0.0, 0.0, 1.0}

[0044] P v2 = {0.0, 0.0, 0.28, 0.0, 0.72}

[0045] P v3 = {0.0, 0.0, 0.26, 0.0, 0.74}

[0046] P v4 = {0.0, 0.0, 0.19, 0.0, 0.81}

[0047] P v5 = {0.0, 0.0, 0.29, 0.0, 0.71}

[0048] P v6 = {0.0, 0.45, 0.14, 0.0, 0.41}

[0049] CPU 106 further includes a quantization signature generation module 204 for generating a quantization signature for each scan action determined by classifier 202. A scan action is a user action performed to scan an item in the scan area of a self-checkout (SCO) terminal.

[0050] The quantization signature generation module 204 includes a data filtering and binarization module 205, a silent interval detection module 206, and a compressed word synthesis module 207.

[0051] The data filtering and binarization module 205 is configured to apply a filter to the category probability vectors generated by classifier 202 to minimize the errors of classifier 202. If classifier 202 uses a single category to classify the continuous movement on the scanner for an entire sequence except for one isolated frame, then classifier errors will occur. In this case, the isolated frame may be misclassified.

[0052] The following is an example output of the probability vectors of six consecutive image frames in a video stream from classifier 202, where the six consecutive image frames cover the continuous movement on the scanner. For image frame in For each probability vector P vn includes the classification probability of the image frame in each of the five categories of "hand", "object in hand", "object", "body part", and "empty scanner".

[0053] P v1 = {0.0, 0.0, 0.28, 0.0, 0.72}

[0054] P v2 = {0.0, 0.0, 0.28, 0.0, 0.72}

[0055] P v3 = {0.0, 0.0, 0.01, 0.27, 0.72}

[0056] P v4 = {0.0, 0.0, 0.28, 0.0, 0.72}

[0057] P v5 = {0.0, 0.0, 0.28, 0.0, 0.72}

[0058] P v6 = {0.0, 0.0, 0.28, 0.0, 0.72}

[0059] It can be clearly seen that there are differences in the probability vector P of the third image frame of the video sequence, which means that the classifier 202 makes an error in the classification of the third image frame. The data filtering and binarization module 205 corrects the error in the classification of the third image frame based on the substantially similar information covered by these six frames. In one embodiment of the present disclosure, the data filtering and binarization module 205 corrects the error by removing the error frame. v3 Then, the data filtering and binarization module 205 is configured to binarize the filtered value of the probability vector using a heuristic threshold, whereby if each component of the probability vector is equal to or greater than the heuristic threshold, a value of "1" is assigned to it, otherwise a value of "0" is assigned to it.

[0060] In one example, when the heuristic threshold is 0.2, the filtered probability vector P for an example of five consecutive image frames

[0061] can be represented as follows:[[]] vf can be represented as follows:

[0062] P vf1 = {0.0, 0.0, 0.0, 0.0, 1.0}

[0063] P vf2 = {0.0, 0.0, 0.28, 0.0, 0.72}

[0064] P vf3 = {0.0, 0.0, 0.26, 0.0, 0.74}

[0065] P vf4 = {0.0, 0.0, 0.39, 0.0, 0.71}

[0066] P vf5 = {0.0, 0.45, 0.14, 0.0, 0.41}

[0067] And the corresponding binarized probability vector P vb Can be as follows:

[0068] P vb1 = {0 0 0 0 1}

[0069] P vb2 = {0 0 1 0 1}

[0070] P vb3 = {0 0 1 0 1}

[0071] P vb4 = {0 0 1 0 1}

[0072] P vb5 = {0 1 0 0 1}

[0073] Therefore, each binarized probability vector P vb Is a binarized string of binary digits that can be used to determine class transitions in consecutive frames. For example, in the first image frame, the binary value corresponding to the class "object" is "0", and in the second image frame, the binary value corresponding to the class "object" is "1", which means there is an obvious class transition from the first to the second image frame. Similarly, in the fourth image frame, the binary value corresponding to the class "object in hand" is "0", and the binary value corresponding to the class "object" is "1". In the fifth frame, the binary value for "object in hand" becomes "1", and the binary value for "object" becomes "0". This clearly indicates that the user picked up the object during the transition from the fourth frame to the fifth frame. Therefore, the binarized / quantized probability vector provides information about class transitions in consecutive image frames.

[0074] The silence interval detection module 206 is configured to detect one or more silence intervals in the video stream. In one embodiment of the present disclosure, during a silence interval, no activity is detected in the scanning area within a threshold duration. In one example, the threshold duration can be set to "0.5 seconds", and when the binary value of the category "empty scanner" of the corresponding image frame is always "1" in a time interval greater than 0.5 seconds, the entire time interval is marked as a "silence interval".

[0075] The compressed word synthesis module 207 is configured to generate a compressed word sequence based on the binarized string generated by the data filtering and binarization module 205. Compressed words are generated based on the category transitions from "1" to "0" and from "0" to "1" in consecutive image frames.

[0076] In one embodiment of the present disclosure, each word consists of 2*N letters associated with the semantics of process actions, where the letters are from the alphabet and N represents the number of categories. In one example, if the number of categories is 5, then each word consists of 10 letters. For each category, the transition from "0→1" generates a specific "start" letter (e.g., "O" for the category "object"), and the transition from "1→0" generates an "end" letter (e.g., "o" for the category "object").

[0077] Thus, the alphabet for the five categories of "hand", "object in hand", "object", "body part", and "empty scanner" contains the following letters:

[0078]

[0079]

[0080] In one embodiment of the present disclosure, two adjacent words are separated by at least one frame classified as "empty scanner". According to the length of consecutive "1" "empty scanner" values, it can indicate whether it is a silence interval.

[0081] An example of the quantized output generated by the compressed word synthesis module 207 is as follows:

[0082] Silence

[0083] OoE

[0084] Silence

[0085] OQoOqBobE

[0086] Silence

[0087] The sequence-dependent classifier 208 is configured to receive the quantized output from the compressed word synthesis module 207 and extract one or more scanning actions from a sequence of successive transitions represented as letters. The sequence-dependent classifier 208 includes a machine learning-based engine, and the engine used here involves an engine that can study algorithms and statistical models and can use these algorithms and models to effectively perform specific tasks without using explicit instructions, relying on patterns and inferences. Examples of the sequence-dependent classifier 208 include recurrent neural networks (RNNs), k-nearest neighbor algorithms (KNNs), and support vector machine (SVM) algorithms, among others.

[0088] The sequence-dependent classifier 208 analyzes the compressed word sequence to identify human activities from the video stream. The compressed word sequence is analyzed to determine different category transitions in the region of interest. This determination of category transitions results in the identification of human activities from the video stream. The sequence-dependent classifier 208 identifies transitions in the binary input signal, indicating basic actions.

[0089] Thus, the quantization signature generation module 204 provides a quantization process for the input signal of the classifier 202 from the region of interest where the observed activity occurs. This transition quantization method aims to use the sequence-dependent classifier 208 to reduce the effects of temporal variations and various body part movements during the activity recognition process.

[0090] Figure 3 is a flowchart showing a method 300 for identifying human activities from a video stream according to an embodiment of the present disclosure. Some steps will be discussed in comparison with Figure 2 the system shown.

[0091] In step 302, an image frame of the video stream is classified by a classifier into one or more categories in a predefined set of categories, where the image frame is classified based on user actions in the region of interest of the image frame. In one embodiment of the present disclosure, the classifier is a convolutional neural network. In another embodiment of the present disclosure, the predefined set of categories for a self-checkout (SCO) scanning area includes categories such as hands, objects in hands, objects, body parts, and empty scanners.

[0092] In step 304, a category probability vector is generated for the image frame based on the classification, where the category probability vector includes a set of classification probabilities of the image frame in each predefined category. In one example, the category probability vector Pv is represented as:

[0093] P vf ={P Hand ,P HandObject ,P Object ,P BodyPart ,PEmptyScanner}

[0094] where P Hand = the probability that the image frame is classified into the category "hand"

[0095] P HandObject = the probability that the image frame is classified into the category "object in hand"

[0096] P Object = the probability that the image frame is classified into the category "object"

[0097] P BodyPart = the probability that the image frame is classified into the category "body part"

[0098] P EmptyScanner = the probability that the image frame is classified into the category "empty scanner"

[0099] In step 306, each probability value of the category probability vector is binarized based on a predefined probability threshold. In one example, if each component of the probability vector is equal to or greater than the heuristic threshold, it is assigned a value of "1", otherwise it is assigned a value of "0".

[0100] In step 308, based on the corresponding binarized probability vector, one or more transitions of one or more categories in one or more consecutive image frames of the video stream are determined. For example, if in the first image frame, the binary value corresponding to the category "object" is "0", and in the second image frame, the binary value corresponding to the category "object" is "1", then it means there is an obvious category transition from the first image frame to the second image frame.

[0101] In step 310, a compressed word sequence is generated based on the determined one or more transitions in one or more consecutive image frames. The compressed words are generated based on the transitions from "1" to "0" or from "0" to "1" in the consecutive image frames. In one embodiment of the present disclosure, the compressed words are formed by the letters of an alphabet that contains twice the number of letters as the number of predefined categories. Further, there is at least one inactive frame between each compressed word in the compressed word sequence. In one example, if the number of categories is 5, then each word consists of a total of 10 letters. For each category, the transition "0→1" generates a specific "start" letter (e.g., generating "O" for the category "object"), and the transition "1→0" generates an "end" letter (e.g., generating "o" for the category "object").

[0102] In step 312, one or more user actions are extracted based on the analysis of the compressed word sequences by a sequence-dependent classifier. The one or more user actions can be used to identify human activities in the SCO scan area (scan action area of interest) and to transmit the identification result to the user computing device. In some embodiments, the user computing device can be configured to store or display the identification result. In one embodiment of the present disclosure, the sequence-dependent classifier is a recurrent neural network.

[0103] The present disclosure also relates to a software product recorded on a machine-readable non-transitory data storage medium, where the software product can be executed on computing hardware to implement a method for identifying human activities from a video stream.

[0104] Without departing from the scope of the present disclosure as defined by the appended claims, modifications may be made to the embodiments of the present disclosure described above. The expressions used to describe and claim the present disclosure (e.g., "comprising", "including", "introducing", "consisting of", "having", "is") are intended to be interpreted in a non-exclusive manner, i.e., allowing for items, components or elements that are not explicitly described. References to the singular should also be interpreted as relating to the plural. The numbers within parentheses in the appended claims are intended to assist in understanding the claims and should not be construed as limiting the subject matter claimed by these claims.

Claims

1. A system for identifying human activities from a video stream captured by an imaging device, the system comprising: A memory for storing one or more instructions; And A processor communicatively coupled to the memory to execute the one or more instructions, wherein the processor comprises: A classifier communicatively coupled to the imaging device and configured to: Classify each of a plurality of consecutive image frames of the video stream into one or more categories in a predefined set of categories, wherein the plurality of consecutive image frames are classified based on corresponding regions of interest detected in each of the plurality of consecutive image frames; and Generate a category probability vector for each of the plurality of consecutive image frames based on the classification, wherein the category probability vector comprises a set of classification probabilities of the image frame in each of the predefined categories in the predefined set of categories; A data filtering and binarization module configured to perform filtering and binarization processing on each probability value of the category probability vector based on a predefined probability threshold; A compressed word synthesis module configured to: Determine one or more transitions of the one or more categories in each of the plurality of consecutive image frames based on a comparison of the binarized probability vectors corresponding to each of the plurality of consecutive image frames; and Generate a compressed word sequence based on the determined one or more transitions of the one or more categories in each of the plurality of consecutive image frames, wherein a compressed word is formed by letters in an alphabet, the number of letters in the alphabet being twice the number of predefined categories in the predefined set of categories, and each of the one or more transitions corresponds to a letter of the compressed word; and A sequence-dependent classifier configured to extract one or more user actions by analyzing the compressed word sequence and identify the human activity therefrom.

2. The system according to claim 1, wherein the classifier is a convolutional neural network, and the classifier is trained to classify images in the video stream into each of the predefined categories in the predefined set of categories.

3. The system according to claim 1, wherein the predefined set of categories is for a self-checkout (SCO) scanning area, and the predefined set of categories comprises one or more categories of hand, object in hand, object, body part, and empty scanner.

4. The system according to claim 1, wherein the data filtering and binarization module is further configured to eliminate classifier errors in the category probability vectors of the one or more consecutive image frames.

5. The system according to claim 1, wherein the processor further comprises a silent interval detection module, and the silent interval detection module is configured to detect one or more silent intervals in the video stream based on detection of inactivity in the region of interest within a predefined threshold duration.

6. The system according to claim 1, wherein there is at least one inactivity frame between each of the compressed words in the compressed word sequence.

7. The system according to claim 1, wherein the sequence-dependent classifier is a recurrent neural network.

8. A method for identifying human activities from a video stream, the method comprising: classifying each of a plurality of consecutive image frames of the video stream into one or more categories in a predefined set of categories by a classifier, wherein the plurality of consecutive image frames are classified based on regions of interest detected in each of the plurality of consecutive image frames; generating, based on the classification, a category probability vector for each of the plurality of consecutive image frames, wherein the category probability vector includes a set of classification probabilities of the image frame in each of the predefined categories in the predefined set of categories; performing binarization processing on each probability value of the category probability vector based on a predefined probability threshold; determining one or more transitions of the one or more categories of each of the plurality of consecutive image frames of the video stream based on a comparison of the binarized probability vectors corresponding to each of the plurality of consecutive image frames; generating a compressed word sequence based on the determined one or more transitions of the one or more categories of each of the plurality of consecutive image frames, wherein the compressed word is formed by letters in an alphabet, and the number of letters in the alphabet is twice the number of predefined categories in the predefined set of categories, and each of the one or more transitions corresponds to a letter of the compressed word; and analyzing the compressed word sequence by a sequence-dependent classifier to extract one or more user actions and identify the human activities therefrom.

9. The method according to claim 8, wherein the classifier is a convolutional neural network, and the classifier is trained to classify images in the video stream into each of the predefined categories in the predefined set of categories.

10. The method according to claim 8, wherein the predefined set of categories is for a self-checkout (SCO) scanning area, and the predefined set of categories includes one or more categories of hand, object in hand, object, body part, and empty scanner.

11. The method according to claim 8, further comprising: Eliminating classifier errors in the category probability vectors of the one or more consecutive image frames.

12. The method according to claim 8, further comprising: Detecting one or more silent intervals in the video stream based on inactivity detection in the region of interest within a predefined threshold duration.

13. The method according to claim 8, wherein there is at least one inactivity frame between each of the compressed words in the compressed word sequence.

14. The method according to claim 8, wherein the sequence-dependent classifier is a recurrent neural network.

15. A computer programmable product for identifying human activities from a video stream, the computer programmable product comprising a set of instructions that, when executed by a processor, cause the processor to perform the following processing: Classify each of a plurality of consecutive image frames of the video stream into one or more categories of a predefined set of categories, where the plurality of consecutive image frames are classified based on regions of interest detected in each of the plurality of consecutive image frames; Generate a category probability vector for each of the plurality of consecutive image frames based on the classification, where the category probability vector includes a set of classification probabilities for the image frame in each of the predefined categories in the predefined set of categories; Perform a binarization process on each probability value of the category probability vector based on a predefined probability threshold; Determine one or more transitions of the one or more categories for each of the plurality of consecutive image frames of the video stream based on a comparison of the binarized probability vectors corresponding to each of the plurality of consecutive image frames; Generate a compressed word sequence based on the determined one or more transitions of the one or more categories for each of the plurality of consecutive image frames, where the compressed words are formed from letters in an alphabet that contains twice the number of letters as the number of predefined categories in the predefined set of categories, and each transition in the one or more transitions corresponds to a letter of the compressed word; and Extract one or more user actions by analyzing the compressed word sequence and identify the human activity therefrom.

16. The computer programmable product according to claim 15, wherein at least one inactive frame is included between each of the compressed words in the compressed word sequence.

Images