SYSTEM AND METHOD FOR DETECTING AND DESCRIPTION OF ANOMALIES IN VIDEOS OF A SCENE - Patent application

Abstract

Embodiments of the present disclosure disclose a method and system for video anomaly detection. The system is configured to collect a sequence of input video frames of an input video of a scene. Additionally, the system is configured to segment each input video frame of the sequence of input video frames into a plurality of input video patches. Furthermore, the system is configured to process each of the plurality of input video patches with one or more classifiers. Each of the one or more classifiers corresponds to a deep neural network trained to estimate one or more attributes of the plurality of input video patches from outputs of a penultimate layer of the deep neural network. Furthermore, the system is configured to compare outputs of the penultimate layer. The system is further configured to detect anomalies based on the outputs of the penultimate layer.

Description

[Technical Field]

[0001] FIELD OF THE DISCLOSURE This disclosure relates generally to image processing, and more particularly to detecting anomalies in a video of a scene and providing descriptions of the detected anomalies. [Background technology]

[0002] Closed-circuit television (CCTV) is widely used for security, surveillance, property monitoring, and other purposes. Examples of CCTV applications include observing crime and vandalism in public squares and buildings (such as hospitals and schools), trespassing in restricted areas, monitoring the free flow of road traffic, detecting traffic accidents and queues, and detecting vehicles erroneously driving on one-way streets.

[0003] However, monitoring CCTV displays (by human operators) is a highly labor-intensive task, and there is a significant risk that an event of interest may go unnoticed. This is especially true when operators must simultaneously monitor the outputs of multiple CCTV cameras. As a result, in many CCTV installations, video data is recorded and inspected in detail only when an event is known to have occurred. Even in such cases, the volume of recorded data can be large, making manual inspection of the data cumbersome. Therefore, automated devices are needed to process video images to detect when there is an event of interest. Such detection, referred to herein as video anomaly detection, can be used to bring the event to the operator's immediate attention, trigger actions in response to the anomaly, place index marks in recorded video, and / or trigger selective recording of CCTV data.

[0004] The problem of video anomaly detection is to automatically detect activity in a portion of a video that differs from the activity seen in a normal video of the same scene. For example, the video may be of a street scene where people are walking along the sidewalk. The anomalous activity to be detected may be people fighting or climbing over a fence, or a car driving on the sidewalk.

[0005] Various approaches to the video anomaly detection problem have been studied in the literature. For example, one approach uses a convolutional neural network autoencoder trained on a particular scene or set of scenes to reconstruct frames of a reference video. The idea is that the autoencoder learns to reconstruct normal frames with a small error, but will have a larger reconstruction error for abnormal frames. To detect anomalies, the autoencoder is used to reconstruct frames of a test video. Frames with a large reconstruction error are flagged as abnormal. Another similar approach uses a neural network to perform prediction of future frames of a video from past frames. Again, the basic idea is that the trained neural network can predict future frames for the normal video on which it was trained, but will have a larger reconstruction error for frames with anomalies.

[0006] Another class of work for video anomaly detection is based on tracking the poses (meaning joint positions and angles) of people in videos using human skeletal models. In this approach, a model of normal human skeletal poses and movements is learned from videos containing only normal human activities. Anomalies are detected by noting novel human skeletal poses or movements in test videos. The main drawback of these approaches is that they can only detect anomalies involving humans. Anomalies involving any other object classes cannot be detected by this approach.

[0007] Another previous method for video anomaly detection is described in U.S. Patent 10,824,935. This method uses a function (such as a Siamese neural network) optimized using machine learning to compare video patches. This was applied to the problem of video anomaly detection by first memorizing a set of normal video patches of a scene (obtained from a normal video of the scene) and then using a Siamese neural network to compare test video patches from a new video of the same scene. Test video patches that are dissimilar to the normal video patches are likely to be anomalous. A major drawback of this approach is that the features learned by the Siamese neural network used to compare video patches are not human-interpretable. This means that the method of U.S. Patent 10,824,935 cannot provide an explanation for why a particular anomaly is anomalous. This method only knows that an anomaly has occurred, but cannot explain why.

[0008] None of the previous works provide a general method for detecting any type of anomaly and a human-understandable explanation for each detected anomaly. For example, the system may detect an anomaly because its reconstruction of the current frame has a large error. The system does not know why the reconstruction error is large. It has no explanation for what caused the large reconstruction error.

[0009] Therefore, there remains a need for a system and method for detecting anomalies in an input video that can provide a human-understandable explanation of "why" a particular activity in the scene is anomalous. Summary of the Invention

[0010] To solve the aforementioned problems, an objective of some embodiments is to compare high-level features estimated from a neural network of a normal video with high-level features of an input video from the same scene to detect anomalies. Hereinafter, "normal video" and "reference video" may be used interchangeably to mean the same thing. As used herein, a "reference video" may correspond to a video that includes a set of video frames corresponding to normal activity in a video scene.

[0011] The system divides the input video into multiple spatial regions. A spatial region may be defined by a rectangle with a specific height and width in pixels. The multiple spatial regions may overlap. The reference video is then partitioned into video patches by sliding a three-dimensional window along the time dimension for each spatial region. For example, each video patch includes a spatial dimension equal to the dimension of the spatial region and a time dimension that defines the number of video frames in the video patch.

[0012] Additionally, the system is configured to learn a set of "high-level" attributes or features that represent the appearance and motion characteristics present in the video patch. For example, the high-level attributes can consist of the object classes (cars, people, bicycles, etc.) present in the video patch, the direction of motion for objects moving within the video patch, and the speed of motion for objects moving within the video patch. The high-level attributes can be estimated directly from the video patch using deep neural networks.

[0013] A deep neural network includes an input layer (containing video patches), multiple hidden layers, and an output layer (high-level attributes). A deep neural network has a penultimate layer known as a penultimate layer, whose output consists of high-level feature vectors that are mapped to high-level attributes of a reference video. The term "high-level attributes" is used to mean human-interpretable attributes of a video patch, such as the set of object classes that appear in the video patch or the direction of motion of an object that appears in the video patch. The term "high-level features" is used to mean the internal representation of a classifier or deep neural network trained using machine learning. The high-level features can be mapped to high-level attributes using a classifier or deep neural network.

[0014] In one embodiment, the system is configured to generate a set of exemplars for each spatial region of the reference video. An exemplar is a set of high-level features from the penultimate layer of a deep neural network trained to estimate high-level appearance or motion attributes. The set of exemplars for a particular spatial region in the reference video represents all high-level feature vectors found in that spatial region, given all video patches that occur in that spatial region in the reference video. In other words, the set of exemplars for a spatial region represents all of the typical activities that occur in that spatial region in the reference video.

[0015] Furthermore, to detect anomalies in the scene input or test video, the test video is partitioned into the same spatial regions in the same manner as the reference video. Video patches are extracted by scanning along the time dimension for each spatial region, as was done for the reference video. High-level features are computed for each test video patch by a deep neural network. The high-level features are compared to each stored exemplar for the corresponding spatial region. The system is further configured to assign an anomaly score to the test video based on the comparison. The anomaly score assigned to a test video patch is the minimum distance of the high-level features of that video patch to each exemplar for the corresponding spatial region. The anomaly score is smaller if the high-level features are close to at least one exemplar, further indicating that the video patch is free of anomalies. The anomaly score is larger if the high-level features are far from all exemplars, thus indicating that the video patch is anomalous.

[0016] If a video patch is found to be anomalous, the system can provide an explanation by finding which of the high-level features did not sufficiently match those of the closest exemplar. The system provides an explanation by using the final layer of a deep neural network to map those high-level features to corresponding high-level attributes and showing that the test video patch differs from normal activity with respect to the mismatched high-level attributes. For example, if the high-level attributes represent object class and direction of motion, an explanation for the anomaly could be that the test video patch contained an unexpected object or that it contained an object moving in an unexpected direction.

[0017] Accordingly, one embodiment discloses a system for video anomaly detection and explanation. The system includes a processor and a memory. The memory stores instructions that, when executed by the processor, cause the system to collect a sequence of input video frames of an input video of a scene. The system is further configured to segment each input video frame of the sequence of input video frames into a plurality of input video patches. Each of the plurality of input video patches is a spatiotemporal patch. The system is further configured to process each of the plurality of input video patches with one or more classifiers. Each of the one or more classifiers corresponds to a deep neural network, the deep neural network having an output layer trained to estimate one or more attributes of the plurality of input video patches from an output of a penultimate layer of the deep neural network. The system is further configured to compare an output of the penultimate layer of the one or more classifiers, generated using the plurality of input video patches, with a corresponding reference output of the penultimate layer of the one or more classifiers, generated using a corresponding reference video patch. A reference video patch of the corresponding reference video patch and an input video patch of the plurality of input video patches correspond to the same spatial region. The output of the penultimate layer of a particular classifier of the one or more classifiers processing the corresponding reference video patch and the input video patch corresponds to one another. Additionally, the system is configured to detect an anomaly when the output of the penultimate layer of the particular classifier is dissimilar to the corresponding reference output of the penultimate layer of the particular classifier. The system is also configured to provide an output including a description of the type of anomaly detected. The output is provided based on the dissimilarity of the penultimate layer output between the input video patch and the closest matching reference video patch from the same spatial region.

[0018] To that end, each of the plurality of input video patches has a spatial dimension that defines the spatial region of the spatiotemporal patch in each of the sequence of input video frames, and a temporal dimension that defines the number of input video frames that form the spatiotemporal patch.

[0019] In some embodiments, the reference video patches are generated by segmenting a training sequence of reference video frames, where the reference video corresponds to a video of normal activity occurring in the same scene as the input video.

[0020] To do so, the deep neural network is trained using videos that are different from the reference videos used to learn the exemplar-based model of a particular scene. The source of the reference videos for the deep neural network can be from surveillance cameras installed in one or more locations.

[0021] In some embodiments, to streamline the comparison, the spatiotemporal division of the input video is identical to the spatiotemporal division of the reference video.

[0022] To that end, the system is configured to compare outputs generated by the penultimate layer of the one or more classifiers with corresponding reference outputs using one or more algorithms associated with nearest neighbor search, where the one or more algorithms correspond to brute force search, kd-trees, k-means trees, locality sensitive hashing, etc.

[0023] In some embodiments, the one or more attributes of the input video patch include appearance and motion attributes, such as object class, direction of motion for objects in the input video patch, speed of motion in each direction, and size of moving objects in the input video patch.

[0024] Accordingly, another embodiment discloses a method for performing video anomaly detection. The method includes collecting a sequence of input video frames of an input video of a scene. The method further includes partitioning the sequence of input video frames into a plurality of input video patches. Each of the plurality of input video patches is a spatiotemporal patch defined in space and time. The method includes processing each of the plurality of input video patches with one or more classifiers. Each of the one or more classifiers corresponds to a deep neural network, the deep neural network having an output layer trained to estimate one or more attributes of the plurality of input video patches from an output of a penultimate layer of the deep neural network. The method further includes comparing an output of the penultimate layer of the one or more classifiers generated using the plurality of input video patches with a corresponding reference output of the penultimate layer of the one or more classifiers generated using a corresponding reference video patch. The reference video patch of the corresponding reference video patch and an input video patch of the plurality of input patches correspond to the same spatial region. The output of the penultimate layer of a particular classifier of the one or more classifiers processing the corresponding reference video patch and input video patch corresponds to each other. The method also includes detecting an anomaly when an output of the penultimate layer of the particular classifier is dissimilar to a corresponding reference output of the penultimate layer of the particular classifier. The method includes providing an output including a description of the type of anomaly detected. The output is provided based on one or more attributes of the input video patch estimated by the output layer of the particular classifier using the dissimilarity output of the penultimate layer of the closest matching reference video patch. [Brief explanation of the drawings]

[0025] [Figure 1A] 1 is a flowchart illustrating detecting anomalies in an input video of a scene, according to various embodiments. [Figure 1B]1 illustrates a block diagram of an environment for detecting anomalies in an input video of a scene, according to various embodiments. [Figure 1C] 1 shows a block diagram of a system for detecting anomalies in an input video of a scene, according to various embodiments. [Figure 2] 1 illustrates an example of partitioning a sequence of input video frames into a set of spatiotemporal patches, according to various embodiments. [Figure 3] 1 is an example architecture of a deep neural network used in anomaly detection, according to various embodiments. [Figure 4A] FIG. 1 shows a diagram of a set of deep neural networks for estimating high-level attributes from a single video patch, in accordance with various embodiments. [Figure 4B] FIG. 1 shows a schematic diagram for selecting a set of exemplars using the output of one or more deep neural networks evaluated on a set of reference video patches, according to various embodiments. [Figure 5] 1 illustrates a schematic diagram of a nearest neighbor search algorithm for finding exemplars that are closest to high-level features of an input video patch, according to various embodiments. [Figure 6] 1 illustrates a flowchart of a method for anomaly detection, according to various embodiments. [Figure 7] 1 shows a block diagram of a computer-based system for detecting anomalies in an input video, according to various embodiments. [Figure 8A] 1 illustrates an example use of a system to detect a person riding a bicycle as an anomaly on a street sidewalk, according to various embodiments. [Figure 8B] 1 illustrates an example use of a system to detect a person riding a bicycle as an anomaly on a street sidewalk, according to various embodiments. [Figure 9A] 1 illustrates an example of the use of the system for detecting traffic law violators in a street scene, according to some embodiments. [Figure 9B] 1 illustrates an example of the use of the system for detecting traffic law violators in a street scene, according to some embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0026] In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the present disclosure. However, it will be apparent to those skilled in the art that the present disclosure may be practiced without these specific details. In other instances, devices and methods are shown only in block diagram form to avoid obscuring the disclosure. It is contemplated that various changes may be made in the function and arrangement of elements without departing from the spirit and scope of the disclosed subject matter as set forth in the appended claims.

[0027] As used in this specification and claims, the words "for example," "for example," "e.g.," "etc.", and the verbs "comprise," "have," "include," and other verb forms thereof, when used in conjunction with a list of one or more components or other items, should each be construed as open-ended, meaning that the list should not be viewed as excluding other additional components or items. The word "based on" means based at least in part on. Furthermore, it should be understood that the phraseology and terminology used herein are for purposes of description and should not be considered limiting. Any headings used within this description are for convenience only and have no legal or limiting effect.

[0028] In the following description, specific details are given for a thorough understanding of the embodiments. However, those skilled in the art will understand that the embodiments may be practiced without these specific details. For example, systems, processes, and other elements in the disclosed subject matter may be shown as components in block diagram form so as not to obscure the embodiments in unnecessary detail. In other instances, well-known processes, structures, and techniques may be shown without unnecessary detail to avoid obscuring the embodiments. Furthermore, like reference numbers and names in the various drawings indicate like elements.

[0029] An objective of some embodiments is to perform anomaly detection using a system that utilizes a deep neural network to divide an input video into overlapping spatial regions and compare each of the spatial regions of the input video to a corresponding region in a reference video. Deep neural network-based systems are capable of processing complex data inputs. Such systems "learn" to perform tasks by considering examples, generally without being programmed with task-specific rules. To that end, it would be advantageous to provide such a system for direct comparison of activity in the input video and the reference video for automated anomaly detection.

[0030] Furthermore, it is an objective of some embodiments to provide human-interpretable explanations for anomaly detection. The system learns a set of "high-level" attributes that represent the appearance and motion attributes present in the video patch. For example, the high-level attributes can consist of the object classes (cars, people, bicycles, etc.) present in the video patch, the direction of motion for objects moving within the video patch, and the speed of motion for objects moving within the video patch. The high-level attributes are estimated directly from the video patch using deep neural networks (or other classification or regression methods). System Overview

[0031] FIG. 1A shows a flowchart 100A illustrating steps for detecting anomalies in an input video of a scene, according to various embodiments. In step 101, one or more deep neural networks are trained using video captured from various sources to estimate high-level features of objects and object motion. In one example, the objects include, but are not limited to, cars, people, bicycles, trees, houses, dogs, etc. Generally, deep neural networks have various network architectures but always include a penultimate (penultimate) layer that outputs a feature vector that is mapped to an output layer. The output layer represents high-level features that are human-interpretable and serve as descriptors for the appearance and motion content of the video. The one or more deep neural networks are trained only once and are not dedicated to a specific scene.

[0032] In step 103, a reference video of the scene is fetched from the image capture device 107, and an exemplar-based model is constructed. The exemplar-based model is a model of typical activity in each spatial region of the reference video using high-level features estimated by one or more trained deep neural networks. The reference video of the scene is captured by a stationary image capture device 107. The reference video is partitioned into possibly overlapping spatial regions, and for each spatial region, a fixed long-term window is slid along the time dimension to generate video patches. Furthermore, the trained one or more deep neural networks calculate a set of high-level feature vectors for each video patch. A subset of the set of high-level feature vectors for each video patch of a spatial region is selected as the exemplar set for that spatial region. The set of high-level feature vectors for each video patch forms the exemplar-based model of the reference video.

[0033] In step 105, the exemplar-based model constructed in step 103 is utilized to detect anomalies in an input video of a scene captured by the same stationary imaging device 107 used in step 103. The input video is partitioned into the same spatial regions used for the reference video in step 103. A fixed long-term window is slid along the time dimension to generate a video patch for each spatial region. For each video patch, high-level features for the given input video are calculated using one or more deep neural networks. Furthermore, the high-level features for the given input video are compared with the high-level features calculated for the corresponding region of the reference video to generate output 109. In one example, for each video patch, one or more deep neural networks are used to calculate a set of high-level feature vectors. For each set of high-level feature vectors, a closest exemplar is found, and the distance between the set of high-level feature vectors of the input video patch and the closest exemplar is calculated. If the distance is greater than a threshold, an anomaly is detected as output 109. Anomaly explanations are also formed by computing high-level features, finding high-level features with larger distances, and mapping these to unmatched high-level features of high-level attributes, whose high-level attributes indicate anomalous parts of the video patch.

[0034] 1B illustrates an environment 100B for detecting anomalies in an input video, in accordance with various embodiments. The environment 100B includes a system 102, a sequence of input video frames 108, a sequence of reference video frames 110, and an image capture device 107.

[0035] System 102 includes processor 104 and memory 106. Memory 106 includes at least one of RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical disk storage, magnetic cassette, magnetic tape, magnetic disk storage, or any other storage medium usable to store desired information and accessible by system 102. Memory 106 may include non-transitory computer storage media in the form of volatile and / or non-volatile memory. Memory 106 may be removable, non-removable, or a combination thereof. Exemplary memory devices include solid-state memory, hard drives, optical disk drives, etc. Memory 106 stores instructions executed by processor 104. Execution of the instructions by processor 104 causes system 102 to perform a set of actions described below.

[0036] System 102 is configured to collect a sequence 108 of input video frames of an input video of a particular scene. The input video is received from an image capture device 107. Image capture device 107 includes, but is not limited to, a video camera. Image capture device 107 is a stationary video capture device. In one embodiment, the input video is received in real time from image capture device 107. Image capture device 107 may be located anywhere in environment 100B or may be connected to system 102 via a communications network.

[0037] The system 102 segments each input video frame of a sequence 108 of input video frames of an input video into a plurality of input video patches. The system is further configured to receive a sequence 110 of reference video frames from the same imaging device 107 from which the input video is received. The sequence 110 of reference video frames includes one or more reference videos. Each reference video of the one or more reference videos corresponds to a video of normal activity occurring at the same location as the input video. The reference videos are processed to build an exemplar-based model of normal activity (as described in FIG. 1A ). The system 102 is configured to process the plurality of input video patches and compare each of the plurality of input video patches to exemplars learned from the reference video to detect anomalies in the input video. The system 102 is further configured to provide an output 109. The output 109 includes a detection of the anomaly and a description of the type of anomaly detected.

[0038] 1C shows a block diagram 100C of a system 102 for detecting anomalies in an input video of a scene, according to various embodiments. The system 102 includes a processor 104 and a memory 106. The processor 104 is associated with the memory 106 (as described above in FIG. 1A). The memory 106 stores instructions that are executed by the processor 104. Execution of the instructions by the processor 104 causes the system 102 to perform a set of actions, which are described below.

[0039] The processor 104 of the system 102 includes a collection module 116, a segmentation module 118, one or more classifiers (or equivalently, deep neural networks) 120, a comparison module 124, and a detection module 126. The system 102 collects a sequence of input video frames 108 of an input video through the convenience of the collection module 116. The collection module 116 is configured to collect the sequence of input video frames. For example, the collection module 116 may be communicatively coupled to or include an I / O interface for receiving and / or transmitting data input to the system 102. The collection module 116 further sends the sequence of input video frames to the segmentation module 118. The segmentation module 118 is configured to segment the sequence of input video frames 108 into a plurality of input video patches. Each of the plurality of input video patches is a spatiotemporal patch (as described in detail in FIG. 2 ).

[0040] Each input video patch of the plurality of input video patches is then sent to one or more classifiers 120. The one or more classifiers 120 process each of the plurality of input video patches. Each of the one or more classifiers 120 corresponds to a deep neural network having an output layer 122. The output layer 122 of the deep neural network is trained to estimate one or more attributes of the plurality of input video patches from the output of the penultimate layer of the deep neural network.

[0041] Generally, a neural network includes an input layer, one or more hidden layers, and an output layer. The output layer 122 is calculated from the penultimate layer of the deep neural network to which it is connected. Furthermore, the penultimate layer is the last hidden layer of the deep neural network. The output from the penultimate layer corresponds to a feature vector that can be mapped to one or more attributes of a video patch. The neural network is trained so that its output layer 122 estimates one or more attributes of its input video patch. The one or more attributes of an input video patch of the multiple input video patches correspond to appearance and motion attributes. The appearance and motion attributes include at least one of an object class present in the multiple input video patches, a direction of motion for an object in each of the multiple input video patches, a speed of motion in each direction, and a size of a moving object in the multiple input video patches.

[0042] Additionally, the system 102 utilizes a comparison module 124. The comparison module 124 is configured to compare the output of a penultimate layer of the one or more classifiers 120, generated using a plurality of input video patches, with a corresponding reference output of the penultimate layer of the one or more classifiers 120, generated using training on a plurality of reference video patches. Each of the plurality of reference video patches and the input video patch of the plurality of input video patches correspond to the same spatial region of the scene. The corresponding reference video patch and the output of the penultimate layer of a particular classifier of the one or more classifiers 120 processing the input video patch correspond to each other. The plurality of reference video patches for each spatial region are a subset of all possible video patches in the sequence of reference video frames 110 and are selected to cover the entire set of reference video patches.

[0043] The comparison of the output of the penultimate layer of one or more classifiers 120, generated using a plurality of input video patches, with the corresponding reference output of the penultimate layer using a plurality of reference video patches is utilized by the detection model 126. The detection model 126 is configured to detect an anomaly when the output of the penultimate layer of a particular classifier of the one or more classifiers 120 is dissimilar to the corresponding reference output of the penultimate layer of the particular classifier.

[0044] Additionally, the detection model 126 provides an output 109. The output 109 includes a description of the type of anomaly detected. The output 109 is provided based on one or more attributes of the input video patch estimated by the output layer of the particular classifier using the dissimilarity output of the penultimate layer of the particular classifier to the closest matching exemplar. In one example, "The input video contains a person riding a bicycle at location A, but the reference video never contained a person riding a bicycle at location A." The detection model 126 detects the anomaly in the input video and provides the description, "The anomaly is detected by the presence of a person riding a bicycle at location A, which is unusual."

[0045] 2 illustrates an example spatiotemporal partitioning 200a of a sequence of video frames of a video 202 into a set of spatiotemporal patches 204, according to various embodiments. The sequence of video frames of the video 202 is provided as input to the partitioning module 118. The partitioning module 118 partitions each video frame of the sequence of video frames into a set of spatiotemporal patches 204. Each spatiotemporal patch, e.g., video patch 206, is defined in space and time by a spatial dimension 208 that defines the area of ​​the spatiotemporal patch within each video frame and a temporal dimension 210 that defines the number of video frames that form the spatiotemporal patch 206. The set of spatiotemporal patches 204 overlap in the spatial dimension 208 and the temporal dimension 210. In one embodiment, the video 202 is an input video. In a second embodiment, the video 202 is a reference video. As used herein, a reference video patch and an input video patch of the same area correspond to each other.

[0046] Various embodiments use different spatiotemporal partitions of the video 202 to define a set of spatiotemporal patches 204. The set of spatiotemporal patches 204 is then utilized by a deep neural network, such as the deep neural network 120 shown in FIG. 1B, to estimate one or more attributes of each spatiotemporal patch in the set of spatiotemporal patches 204. However, in various implementations, the spatiotemporal partitions of the input video are the same as the spatiotemporal partitions of the reference video, so that the outputs of the penultimate layers of the one or more classifiers 120 are comparable.

[0047] Each of the multiple input video patches is processed using one or more deep neural networks 120. The one or more deep neural networks 120 extract high-level features from each of the multiple input video patches. The high-level features correspond to one or more attributes, such as object class, direction of motion, and speed. The one or more attributes mentioned are for illustrative purposes only and should not be construed as limiting the scope of the present disclosure in any way. Some embodiments are based on the recognition that detecting anomalous portions of any input video can be achieved by comparing the high-level features of each of the multiple input video patches (i.e., each spatiotemporal region) with all the high-level features of a normal reference video within the same spatial region. Comparing the input video with a normal reference video within the same spatial region is important because normal activity is location-dependent. For example, a person walking along a sidewalk is normal, but a person walking in the middle of a street or on top of a building is usually abnormal.

[0048] In one example, the system 102 is configured to assign an anomaly score to an input video or a test video. In one example, the anomaly score assigned to a test video patch is the minimum distance of the high-level features of that video patch to each of the exemplars for the corresponding spatial region. One exemplar is a set of high-level features from the penultimate layer of a deep neural network 120 trained to estimate high-level appearance or motion attributes. The set of exemplars for a particular spatial region in a normal video represents all high-level feature vectors found in that spatial region, considering all video patches that occur in that spatial region in the reference video. In other words, the set of exemplars for a spatial region represents all of the reference activities that occur in that spatial region in the reference video. If the high-level features of the input video are close to at least one exemplar (the high-level features of the reference video) and further indicate that the given input video is free of anomalies, the anomaly score is smaller. If the high-level features of the input video are far from all exemplars (the high-level features of the reference video), thus indicating that the input video is anomalous, the anomaly score is larger.

[0049] FIG. 3 illustrates an example of a deep neural network 120 estimating high-level attributes of an input video patch 310, according to various embodiments. The deep neural network 120 is trained to estimate high-level attributes 340, including, but not limited to, appearance attributes and motion attributes from the spatiotemporal patch. As an example, the high-level attributes may consist of a vector of probabilities representing the likelihood that each of a set of fixed object classes is present in the spatiotemporal patch. As another example, the high-level attributes correspond to a vector representing the fraction of pixels in the spatiotemporal patch that move in each of a set of fixed directions. The deep neural network 120 takes the spatiotemporal video patch 310 as input and outputs a vector representing the high-level attributes 340. The deep neural network 120 has a body 320. The body 320 of the deep neural network 120 may consist of multiple initial layers, including 2D convolutional layers, 3D convolutional layers, pooling layers, nonlinear activation layers, fully connected layers, etc. The result of the initial layers is a penultimate feature vector 330. The penultimate feature vector 330 represents some high-level features of the spatiotemporal video patch 310. The penultimate feature vector 330 is mapped to high-level attributes, which are human-understandable attributes 340, through a final fully connected layer (computed by simple matrix multiplication). The deep neural network 120 is trained using videos that may be involved in video anomaly detection, captured from various sources and independent of their scenes. The deep neural network 120 is trained to estimate high-level attributes present in any spatiotemporal video patch.

[0050] Some embodiments recognize that if the high-level features of an input video patch 310 can be mapped to one or more human-understandable attributes, the system 102 can provide a human-understandable explanation for its decision. Furthermore, the penultimate feature vector 330 of the deep neural network 120 is used as the high-level feature. Furthermore, the explanation consists of high-level attributes corresponding to the high-level features of the input video patch that have a large distance to the closest matching exemplar in the set of exemplars learned from the reference video for the same spatial region of the input video patch. In one embodiment, the penultimate feature vectors 330 computed for two different video patches from the deep neural network 120 are compared using Euclidean distance.

[0051] 4A and 4B show block diagrams for estimating a set of high-level attributes of a single video patch 408 from one or more deep neural networks 120. In one example, five different deep neural networks are shown, such as deep neural network 406, deep neural network 410, deep neural network 412, deep neural network 414, and deep neural network 416. Each deep neural network outputs a different high-level attribute of the single video patch 408. The one or more deep neural networks 120 include an appearance network 406, an orientation network 410, a velocity network 412, a background portion network 414, and a background classifier network 416, etc.

[0052] In this example, the appearance network 406 corresponds to an object class deep neural network that outputs 404 the likelihood that an input video patch 408 contains one of eight object classes, such as a person, a car, a bicycle, a dog, a tree, a house, a skyscraper, or a bridge.

[0053] Additionally, the direction network 410 is trained to output a motion direction histogram 418. The motion direction histogram is an optical flow histogram consisting of 12 bins, each storing the fraction of pixels in the video patch 408 that are estimated to be moving in one of 30 degrees of motion directions.

[0054] The velocity network 412 is trained to output a directional velocity vector 420, which is a vector of average pixel velocities in each direction of motion. The directional velocity vector 420 consists of the average velocity (in pixels per frame) of all pixels that fall into each of the 12 flow histogram bins.

[0055] The background portion network 414 is trained to output a portion of static pixels 422. The portion of static pixels 422 in the video patch 408 gives the approximate size of moving objects in the video volume 408.

[0056] The background classifier network 416 is trained to output a background classification 424 for the video patch 408. The background classification classifies whether the video volume 408 has motion or not.

[0057] The high-level features, such as features 402, output from each deep neural network result in a set of high-level features.

[0058] FIG. 4B shows a block diagram 400B for selecting a set of exemplars to model activity present in a reference video frame. One exemplar is a set of high-level feature vectors computed from a single video patch using the penultimate layer output of one or more previously trained deep neural networks 120. Once the deep neural networks 120 are trained, the system 102 utilizes them to represent the reference video. As shown in FIG. 4B, to process each of multiple reference video frames 426, a spatiotemporal patch of dimensions [h×w×t] with spatial stride (r, c) and temporal stride s is defined to construct video volumes 408a, 408b.

[0059] In one example, it is assumed that h = w, where h is approximately the height in pixels of a person in a particular dataset. For each video volume 408 a, 408 b, the system 102 extracts high-level features using a previously trained deep neural network, such as deep neural network 406, deep neural network 410, deep neural network 412, deep neural network 414, and deep neural network 416 (hereinafter also referred to as deep neural networks 406-416). The system 102 concatenates the feature vectors 428 extracted from the penultimate layers of the deep neural networks 406-416 to create a combined feature vector 430 for each video volume 408 a, 408 b in the reference video frame 426.

[0060] The feature vector 428 extracted from the penultimate layer of the trained deep network 406-416 is denoted as "F" to denote the combined feature vector 430, where "app," "ang," "mag," and "bkg" are used to denote the appearance attribute, angle attribute, magnitude attribute, and background pixel subfeature vector, respectively, of size 1x128. Finally, "cls" denotes the binary background classification of size 1x1. F has a size of 1x513.

[0061] After computing the feature vectors 430, the system 102 utilizes an exemplar selection technique using an exemplar selector 432 to create a domain-specific, compact model of the reference data. For each domain, the system 102 uses the following greedy exemplar selection algorithm: 1.) add the first feature vector to the exemplar set; 2.) for each subsequent feature vector, calculate its distance to each feature vector in the exemplar set and add it to the exemplar set only if all distances exceed a threshold th. To compute the distance between two feature vectors F1=[app1;ang1;mag1;bkg1;cls1] and F2=[app2;ang2;mag2;bkg2;cls2], the system 102 uses the L2 distance of corresponding components normalized by a constant to make the maximum distance for each component approximately 1.

[0062] In one example, when the video volume has no motion, the motion component becomes meaningless and the motion feature vector (ang, mag, bkg) is set to a zero vector.

[0063] The distance function can be written as:

number

[0064] Normalization factor Z app , Z ang , Z mag and Z bkg is computed once by finding the maximum L2 distance between a large set of feature vector components.

[0065] 5 shows a schematic diagram 500 of a nearest neighbor search algorithm 530 for finding the exemplars that are closest to the high-level features of an input video patch, according to various embodiments of the present disclosure. v is the high-level feature vector 510 of the input video patch, and each xi (x1, x2, x3, ..., x n) 520 is the high-level feature vector of the reference video patch (exemplar). The nearest neighbor search algorithm 530 calculates f v 510 and nearest neighbor x i 520. If the minimum distance d540 is greater than the anomaly score threshold, the input video patch is declared to have an anomaly.

[0066] Different embodiments use different distance functions between high-level features. In one embodiment, a Euclidean distance function is utilized. The nearest neighbor search algorithm may include different search algorithms. For example, one embodiment may use a distance function for each input feature vector f v 510 for each reference feature vector x i 520, a brute force search algorithm is used. In some implementations, nearest neighbor search algorithm 530 is an approximate nearest neighbor search algorithm, which is not guaranteed to find the minimum distance, but may instead find a feature vector that is close to the minimum. Various nearest neighbor search algorithms known in the art can be used, such as kd trees, k-means trees, and locality sensitive hashing.

[0067] FIG. 6 shows a flowchart 600 of a method for anomaly detection, according to various embodiments. The method may be executed by the processor 104 of the system 102 according to instructions stored in the memory 106. At step 610, the method 600 includes partitioning an input video into input video patches 615. The input video patches 615 correspond to the plurality of video patches described above in FIGS. 1A, 1B, and 2. A video patch is a spatiotemporal region that may be defined by a bounding box within video frames that defines a spatial extent and a fixed number of frames that define a temporal extent (as described above in FIG. 2). Thus, pixels of the video within a spatiotemporal region comprise a video patch. Different video patches may overlap. The union of all video patches covers the entire video sequence.

[0068] At step 620, method 600 estimates high-level features 627 for each of the input video patches using one or more neural networks 625. The one or more neural networks 625 correspond to the deep neural network 120 of FIG. 1C. The one or more neural networks 625 are trained to estimate high-level appearance and motion attributes of each of the input video patches 615, as described above in FIGS. 3, 4A, and 4B. At step 630, each of the high-level features 627 is compared to high-level features 635 of a reference video patch (called an exemplar) from the same spatial region as the high-level features from the input video patch. The corresponding exemplar is retrieved from memory 106 and compared to the high-level features of the input video patch using a distance function, such as the Euclidean distance between each high-level feature, as discussed in FIG. 4B.

[0069] At step 640, method 600 includes detecting an anomaly when the high-level features 627 for the input video patch 615 have a large distance to the closest exemplar 635 for that spatial region. Additionally, method 600 detects an anomaly when the distance between the high-level features 627 for the input video patch 615 and the closest one of the exemplars 635 for the corresponding spatial region is greater than a threshold 660. At step 645, the method generates spatial and temporal coordinate outputs for the anomalous input video patch. At step 650, for each anomalous video patch, high-level attributes are computed from the anomalous video patch's high-level feature vector 627 and the closest exemplar 635 using one or more layers of a neural network 625 (as illustrated in FIG. 3 ) to provide a causal explanation for the anomaly. The causal explanation may be due to dissimilarity of the high-level attributes from the input video patch and its closest matching exemplar. At step 655, method 600 provides a causal explanation for the anomaly as a list of high-level attributes identified as the anomaly cause.

[0070] In this manner, some embodiments may provide an anomaly detector suitable for direct comparison of activity with realistic computational and memory requirements suitable for, for example, a closed circuit television (CCTV) system.

[0071] 7 shows a block diagram of a computer-based system 700 for detecting anomalies in an input video, according to some embodiments. The computer-based system 700 includes a processor 720 configured to execute stored instructions and a memory 740 that stores instructions executable by the processor 720. The processor 720 may be a single-core processor, a multi-core processor, a computing cluster, or any number of other configurations. The memory 740 may include random access memory (RAM), read-only memory (ROM), flash memory, or any other suitable memory system. The processor 720 is connected to one or more input and output devices via a bus 706. The instructions implement a method for detecting anomalies in a video sequence of video frames 795.

[0072] In various embodiments, the computer-based system 700 generates a set of bounding boxes indicating the location and size of any anomalies in each of the video frames 795. The computer-based system 700 is configured to detect anomalies in an input video. First, the computer-based system partitions the video frames 795 into video patches having a fixed height, width, and number of video frames, and then uses a neural network 735 to estimate high-level appearance and motion features 731 (hereinafter, high-level features) for each of the video patches. The high-level features 731 of a video patch of the input video are compared with stored high-level features of reference video patches computed from a reference video of the same scene. When the high-level features 731 computed for an input video patch are dissimilar to all of the high-level reference features for the corresponding spatial region, an anomaly is declared for that video patch. Furthermore, the computer-based system 700 is configured to provide an explanation. The explanation of the anomaly is based on the high-level features 731 of the input video patch that do not match the closest reference feature vector. The explanation includes information about the cause of the anomaly.

[0073] 1A to a keyboard 711 and a pointing device 712. The pointing device 712 may include, but is not limited to, a mouse, a trackball, a touchpad, a joystick, a pointing stick, a stylus, or a touch screen, among others. The computer-based system 700 is linked via a bus 706 to a display interface 760 adapted to connect the computer-based system 700 to a display device 765. The display device 765 may include, but is not limited to, a computer monitor, a camera, a television, a projector, or a mobile device.

[0074] The computer-based system 700 can be connected to an imaging interface 70 adapted to connect the system 102 to an imaging device 775. In one embodiment, video frames 795 of an input video on which anomaly detection is performed are received from the imaging device 775. The imaging device 775 can include a video camera, a computer, a mobile device, a webcam, or any combination thereof.

[0075] In some embodiments, the computer-based system 700 is connected via bus 706 to an application interface 780 adapted to connect the computer-based system 700 to an application device 785. The application device 785 acts based on the results of the anomaly detection. In one example, the application device 785 is a surveillance system that uses the location of the detected anomaly to alert security guards to investigate further.

[0076] A network interface controller 750 is adapted to connect the computer-based system 700 to a network 790 via a bus 706. Through the network 790, video frames 795, e.g., frames of a normal or reference patch of video and / or an input or test patch of video, are downloaded and stored in the computer's storage device 730 for storage and / or further processing. In some embodiments, the reference and input video patches are stored as a set of high-level features or high-level attributes extracted from the corresponding video patches, e.g., an input feature vector (high-level appearance and motion features 731) or a reference feature vector 733. In this manner, storage requirements can be reduced while improving subsequent processing of the video. Examples of high-level attributes extracted from the video may include the class of objects present in the video patch (as described above in FIG. 1B), the direction of object motion, and the speed of object motion. The high-level features may be internal features of a neural network that can be mapped to high-level attributes.

[0077] In one embodiment, computer-based system 700 corresponds to system 102 of Figure 1B. In another embodiment, computer-based system 700 is associated with system 102. In yet another embodiment, computer-based system 700 is part of system 102. Additionally, Figures 8A, 8B, 9A, and 9B illustrate examples of use of system 102.

[0078] 8A and 8B illustrate a use case 800 of the system 102 according to some embodiments. The use case 800 includes a visualization of a region 802 of a street scene and multiple exemplars 806 learned for the region 802 from a reference video of the same scene. The system 102 analyzes the region (test frame) 802 on the sidewalk of the street scene and extracts multiple exemplars 806 for the region 802. The region 802 is a spatial region. The multiple exemplars 806 learned for the region 802 include a static background 806A, backgrounds 806B, 806C, and 806D with little motion, a person 806E moving slowly mainly to the left, unknown objects 806F and 806G with some motion to the left, people 806H and 806I moving slowly to the bottom right, and a mostly static background 806J. The multiple exemplars 806 may not be limited to the examples mentioned.

[0079] In the illustrated test frame (region 802), a bicyclist is riding a bicycle on a sidewalk. A bounding box 804 indicates the bicyclist on the sidewalk. The system 102 generates a visualization 804A of a video volume centered on region 802. Visualization 804A shows that region 802 contains a bicyclist (a high-level attribute) moving quickly down and to the right on the sidewalk. The system 102 maps the high-level features of region 802 to multiple exemplars 806 (806A-806J). The system 102 identifies exemplar 806H as the exemplar closest to visualization 804A of the high-level features of region 802. Exemplary example 806H indicates a person moving slowly down and to the right. The system 102 then finds a distance 808 between the high-level features of region 802 and the closest exemplar 806H. The distance 808 is approximately 2.19, which is greater than the threshold distance. In one example, if distance 908 is greater than the threshold distance, system 102 generates a high anomaly score for region 802, indicating that region 802 is anomalous. Additionally, system 102 provides an explanation for the high anomaly score. The explanation may state, "The closest exemplar 806H shows a person moving slowly in the lower right, but in region 802, instead of a person, there is a person riding a bicycle. Typically, there are no people riding bicycles on sidewalks, and therefore region 802 has a high anomaly score and is anomalous."

[0080] 9A and 9B illustrate a use case 900 of the system 102, according to some embodiments. The use case 900 includes a region 902 of a street scene and a visualization of multiple exemplars 906 learned about the region 902. Additionally, the use case 900 includes a visualization 904AB of a test video volume of the region 902 that explains the reason for an anomaly detection in the region 902.

[0081] Region 902 shows a car traveling down the street lane to the right and traffic rule violators 904A and 904B. The system 102 analyzes region (test frame) 902 of a street scene and extracts multiple exemplars 906 of region 902. Region 902 is a spatial region. The multiple exemplars 906 learned for region 902 include a car 906A that is not moving (because traffic occasionally stops on this part of the road), a car 906B moving down and to the right at high speed (as expected), a mostly static background 906C, an unknown object 906D with slight movement down and to the right, a car 906E moving down and to the right at high speed, unknown objects 906F, 906G, and 906H moving down and to the right at high speed, and cars 906I and 906J moving down and to the right at high speed. The multiple exemplars 906 need not be limited to the mentioned examples.

[0082] Furthermore, one of the video volumes of region 902 includes a person violating a traffic rule (traffic rule violators 904A, 904B), and system 102 generates a high-level feature visualization 904AB for the video volume of region 902, showing the traffic rule violators 904A, 904B. The system maps the high-level features to multiple exemplars 906. The exemplar closest to the video volume is exemplar 906H, which shows an unknown object moving downward and to the right. No exemplars represent anything resembling a person walking to the right with respect to region 902. Furthermore, system 102 is configured to calculate a distance 908 between the high-level features of region 902 and the closest exemplar 906H. Therefore, distance 908 to the closest exemplar is large, indicating an anomaly. Distance 908 is equal to 2.08, which is greater than the threshold distance of 1.8, indicating region 902 is anomalous. Furthermore, system 102 provides an explanation for the detected anomaly. The explanation may state, "The closest exemplar 906H shows an unknown object moving quickly downward and to the right, but instead of an unknown object, there are traffic rule violators 904A, 904B in region 902. Therefore, an anomaly is detected in region 902 due to traffic rule violators 904A and 904B in the street lane." System 902 provides a human-understandable explanation for the anomaly detection.

[0083] Furthermore, embodiments of the disclosed subject matter may be implemented, at least in part, either manually or automatically. Manual or automatic implementations may be performed or at least assisted by machine, hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof. When implemented in software, firmware, middleware, or microcode, the program code or code segments to perform the necessary tasks may be stored on a machine-readable medium. The necessary tasks may be performed by a processor.

[0084] The above-described embodiments of the present disclosure may be implemented in any of numerous ways. For example, the embodiments may be implemented using hardware, software, or a combination thereof. If implemented in software, the software code may be executed on any suitable processor or collection of processors, whether provided on a single computer or distributed across multiple computers. Such a processor may be implemented as an integrated circuit with one or more processors within an integrated circuit component. However, a processor may be implemented using any suitable form of circuitry.

[0085] Also, the various methods or processes outlined herein may be coded as software executable on one or more processors using any one of a variety of operating systems or platforms. Additionally, such software may be written using any of a number of suitable programming languages ​​and / or programming or scripting tools, and may be compiled as executable machine language code or intermediate code that runs on a framework or virtual machine. Typically, the functionality of the program modules may be combined or distributed as desired in various embodiments.

[0086] Also, embodiments of the present disclosure may be embodied as methods, for which examples are provided. Acts performed as part of a method may be ordered in any suitable manner. Thus, embodiments may be constructed in which acts are performed in an order other than that illustrated, including simultaneously performing some acts shown as sequential acts in an exemplary embodiment. Therefore, it is the object of the appended claims to cover all such variations and modifications that are within the true spirit and scope of the present disclosure.

[0087] Although the present disclosure has been described with reference to certain preferred embodiments, it is to be understood that various other adaptations and modifications can be made within the spirit and scope of the present disclosure. It is, therefore, within the scope of the claims to cover all such variations and modifications that are within the true spirit and scope of the present disclosure.

Claims

1. 1. A system for video anomaly detection, comprising: a processor; and a memory having instructions stored thereon, the instructions, when executed by the processor, causing the system to: collecting a sequence of input video frames of an input video of a scene; The instructions further comprise: partitioning the sequence of input video frames into a plurality of input video patches, each of the plurality of input video patches being a spatiotemporal patch; and when executed by the processor, causing the system to: causing each of the plurality of input video patches to be processed by one or more classifiers, each of the one or more classifiers corresponding to a deep neural network, the deep neural network having an output layer trained to estimate one or more attributes of the plurality of input video patches from an output of a penultimate layer of the deep neural network; and the instructions, when executed by the processor, further cause the system to: comparing the outputs of the penultimate layer of the one or more classifiers generated using the plurality of input video patches with reference outputs of the penultimate layer of the one or more classifiers generated using a plurality of reference video patches from corresponding spatial regions; The plurality of reference video patches are extracted from a reference video of the scene, and the instructions, when executed by the processor, further cause the system to: detecting an anomaly when the output of the penultimate layer of the one or more classifiers for an input video patch is dissimilar to the output of the penultimate layer of the one or more classifiers for the plurality of reference video patches from the same spatial region as the input video patch; 1. A system for video anomaly detection, comprising: providing an output including a description of the type of anomaly detected, the output being provided based on the one or more attributes of the input video patch estimated by the output layer of the one or more classifiers being dissimilar to attributes of a closest matching reference video patch.

2. 2. The system of claim 1, wherein each of the plurality of input video patches has a spatial dimension that defines a spatial region of the spatiotemporal patch in each of the sequence of input video frames, and a temporal dimension that defines a number of input video frames that form the spatiotemporal patch.

3. 2. The system of claim 1, wherein the plurality of reference video patches are generated by segmenting one or more video frames of a reference video present in a sequence of reference video frames, the reference video corresponding to a video of normal activity occurring in the same scene as the input video.

4. The system of claim 3 , wherein the plurality of reference video patches for each spatial region is a subset of all possible video patches in the sequence of reference video frames and is selected to cover an entire set of reference video patches.

5. The system of claim 1 , wherein the spatiotemporal divisions of the input video are identical to the spatiotemporal divisions of the reference video, and the identical spatiotemporal divisions are used to streamline the comparison.

6. 2. The system of claim 1, wherein the one or more attributes of the input video patch include appearance and motion attributes, the appearance and motion attributes including at least one of a direction of motion for objects in the input video patch, a speed of motion in each direction, and a size of moving objects in the input video patch.

7. The system of claim 1 , wherein the deep neural network is trained using a sequence of video frames.

8. 10. The system of claim 1, wherein the system is configured to compare the outputs produced by the penultimate layer of the one or more classifiers with corresponding reference outputs using one or more algorithms associated with nearest neighbor searching, the one or more algorithms corresponding to at least one of a brute force search, a k-d tree, a k-means tree, and a locality-sensitive hashing.

9. The system of claim 1 , wherein the system is configured to calculate a distance between the one or more attributes of the input video and a closest matching attribute of a reference video.

10. 1. A computer-implemented method for performing video anomaly detection, comprising: collecting a sequence of input video frames of an input video of a scene; and partitioning the sequence of input video frames into a plurality of input video patches, each of the plurality of input video patches being a spatiotemporal patch defined in space and time, the method further comprising: processing each of the plurality of input video patches with one or more classifiers, each of the one or more classifiers corresponding to a deep neural network having an output layer trained to estimate one or more attributes of the plurality of input video patches from an output of a penultimate layer of the deep neural network, the method further comprising: comparing the outputs of the penultimate layer of the one or more classifiers generated using the plurality of input video patches with corresponding reference outputs of the penultimate layer of the one or more classifiers generated using a plurality of reference video patches; The plurality of reference video patches are extracted from a reference video of the scene, and the method further comprises: detecting an anomaly when the output of the penultimate layer of the one or more classifiers for an input video patch is dissimilar to the output of the penultimate layer of the one or more classifiers for the plurality of reference video patches from the same spatial region as the input video patch; and providing an output including a description of the type of anomaly detected, the output being provided based on the one or more attributes of the input video patch estimated by the output layer of the one or more classifiers being dissimilar to attributes of a closest matching reference video patch.

11. 11. The method of claim 10, wherein each of the plurality of input video patches is a spatiotemporal patch defined in space and time, with a spatial dimension defining a spatial region of the spatiotemporal patch in each of the sequence of input video frames and a temporal dimension defining a number of input video frames that form the spatiotemporal patch.

12. 11. The method of claim 10, wherein the reference video patch is generated by segmenting one or more video frames of a reference video present in a sequence of reference video frames, the reference video corresponding to video of normal activity occurring at one or more locations.

13. The method of claim 10 , wherein the spatiotemporal divisions of the input video are identical to the spatiotemporal divisions of the reference video, and the same spatiotemporal patches are used to streamline the comparison.

14. 11. The method of claim 10, wherein the one or more attributes of the input video patch include at least one of appearance and motion attributes, such as direction of motion for objects in the input video patch, speed of motion in each direction, and size of moving objects in the input video patch.

15. The method of claim 10 , wherein the deep neural network is trained using a sequence of reference video frames.

16. 11. The method of claim 10, wherein the system is configured to compare the outputs produced by the penultimate layer of the one or more classifiers with corresponding reference outputs using one or more algorithms associated with nearest neighbor searching, the one or more algorithms corresponding to at least one of a brute force search, a k-d tree, a k-means tree, and locality-sensitive hashing.

Images