METHOD AND SYSTEM FOR ZERO-SHOT CROSS-DOMAIN VIDEO ANOMALYSIS DETECTION - Patent application

Abstract

An embodiment of the present invention discloses a method and system for performing video anomaly detection by training a neural network. The method includes collecting a video of one or more digital images from a source domain. The method includes obtaining a set of images of a foreground object present in the video. The method further includes training a first neural network to predict frames of the one or more digital images in the video. The first neural network is trained using a future frame prediction module to predict frames of the one or more digital images. The method includes training a second neural network to classify the predicted frames as normal and classify the synthesized pseudo-anomalous frames as anomalous. The method includes performing video anomaly detection based on training the first neural network and the second neural network.

Description

[Technical field]

[0001] FIELD OF THE DISCLOSURE The present disclosure relates to video anomaly detection, and more particularly, to a method and system for zero-shot cross-domain video anomaly detection. [Background technology]

[0002] Unsupervised Video Anomaly Detection (VAD) is widely used in security and surveillance applications compared to supervised or weakly supervised VAD. This is mainly because unsupervised VAD methods do not require training videos containing unusual events that are rare and laborious to annotate. Thus, unsupervised VAD methods use only normal events in the training videos to mark puzzling activities as anomalous during testing by the trained model. Recently, unsupervised VAD methods in cross-domain settings have been introduced. The goal is to perform VAD in a different target domain given video data containing only normal events from the source domain. However, these cross-domain VADs (xVADs) are few-shot scene adaptation based methods that require access to VAD normal data from both source and target domains. This assumption becomes a serious problem when there is no access to the data (videos of normal activities) required for model learning and adaptation. For example, collection or access to video data in public places may not be allowed for third-party companies due to security and privacy concerns. For this reason, current xVAD methods are ineffective because they are highly dependent on accessing at least a small number of target domain training datasets. Existing solutions to the problem of video anomaly detection (VAD) require access to normal training videos taken from both the source and target domains. As an example of a VAD problem, the source and target domains may be a train station and airport scenario. At training time, only normal videos (not including anomalous videos) taken at the station and the airport are available. At test time, the task is to identify anomalies in a test video taken at the airport (target domain), where the test video includes both normal and anomalous videos. Summary of the Invention

[0003] [Technical issues] Therefore, to address the above-mentioned problems, there is a need for an improved xVAD system and method that can overcome the above-mentioned shortcomings. [Problem Solution]

[0004] The present disclosure provides methods and systems for performing video anomaly detection.

[0005] Some embodiments are based on the recognition that there is a need for zero-shot cross-domain video anomaly detection without target domain data.

[0006] To that end, some embodiments are based on the realization of a zero-shot cross-domain video anomaly detection (zVAD) system. The zVAD system collects a video of one or more digital images from a source domain. The zVAD system also obtains a set of images of foreground objects present in the video. The source domain video corresponds to at least one of a non-task-related or task-related video, a normal event-related video, a publicly available video, and a video combining multiple images.

[0007] Some embodiments are based on the recognition that the zVAD system of the present disclosure is configured to train a first neural network to predict frames of one or more digital images in a video. The first neural network is trained using a future frame prediction module to predict frames of the one or more digital images. The future frame prediction module includes a generator and a classifier. The generator includes an encoder, a memory module, and a decoder. The future frame prediction module is associated with an object-aware anomaly synthesis module, which is configured to generate pseudo-anomalous frames and learn features from normal frames for the pseudo-anomalous frames.

[0008] Some embodiments are based on the recognition that the zVAD system of the present disclosure is configured to train a second neural network. The second neural network is trained to classify predicted frames of the digital images of the one or more digital images as normal and to classify predicted frames of the digital images of the one or more digital images having a corrupted portion with an image from the image set of the foreground object as anomalous. The second neural network is trained using a normality classifier module. The normality classifier module generates a plurality of loss functions utilized to train the second neural network. The plurality of loss functions include at least one of a normality loss, a relative normality loss, an attention positive loss, and a relative attention positive loss. The normality classifier module utilizes the predicted frames received from the future frame prediction module and the one or more pseudo anomalous frames generated by the object recognition anomaly synthesis module to predict a probability of normality or anomaly in the video.

[0009] Accordingly, some embodiments disclose a method for performing video anomaly detection by training a neural network. The method includes collecting a video of one or more digital images from a source domain. The method includes obtaining a set of images of a foreground object present in the video. The foreground object corresponds to at least one of a group of people and an individual. The method further includes training a first neural network to predict frames of the one or more digital images in the video. The first neural network is trained using a future frame prediction module that predicts frames of the digital images of the one or more digital images. The method further includes training a second neural network to classify predicted frames of the digital images of the one or more digital images as normal and classify predicted frames of the digital images of the one or more digital images having a corrupted portion with an image from the set of images of the foreground object as anomalous. The second neural network is trained using a normality classifier module. The method further includes performing video anomaly detection based on training the first neural network and the second neural network.

[0010] The present disclosure is further described in the following detailed description by way of non-limiting examples of exemplary embodiments of the present disclosure with reference to the several noted drawings in which like reference numerals represent like parts throughout the several views of the drawings. The drawings illustrated are not necessarily to scale, with emphasis generally being placed upon illustrating the principles of embodiments of the present disclosure. [Brief description of the drawings]

[0011] [Figure 1] FIG. 1 is a schematic block diagram illustrating a training system for video anomaly detection according to some embodiments of the present disclosure. [Figure 2A] FIG. 2 is a block diagram illustrating a training system for training a first neural network according to some embodiments of the present disclosure. [Figure 2B]FIG. 13 is a block diagram for training a second neural network according to some embodiments of the present disclosure. [Figure 2C] FIG. 2 is a block diagram illustrating a normality classifier module according to some embodiments of the present disclosure. [Diagram 3] 1A-1C are diagrams illustrating representations of one or more pseudo-anomalous frames according to some embodiments of the present disclosure. [Figure 4] 2 illustrates an exemplary representation of the operation of the training system shown in FIG. 1 according to some embodiments of the present disclosure. [Diagram 5] FIG. 1 is a block diagram for jointly training a first neural network and a second neural network to perform video anomaly detection according to some embodiments of the present disclosure. [Figure 6] 5 is a flowchart illustrating a method for performing video anomaly detection using the training system disclosed in FIG. 4 according to various embodiments of the present disclosure. [Figure 7] FIG. 1 illustrates a use case for detecting anomalies using a training system, according to some example embodiments of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] [Description of the embodiment] In the following description, for the purpose of explanation, numerous specific details are set forth in order to provide a thorough understanding of the present disclosure. However, it will be apparent to those skilled in the art that the present disclosure may be practiced without these specific details. In other instances, the apparatus and method are shown only in block diagram form to avoid obscuring the present disclosure. Various changes may be made in the function and arrangement of elements without departing from the spirit and scope of the disclosed subject matter as set forth in the appended claims.

[0013] As used in this specification and claims, terms such as "for example," "for example," and "such as," as well as the verbs "comprise," "have," "include," and each of their other verb forms, when used in conjunction with a list of one or more components or other items, are to be construed as open-ended, meaning that such list should not be considered to exclude other additional components or items. The term "based on" means based at least in part on. Furthermore, it is to be understood that the words and terms used herein are for descriptive purposes and should not be considered limiting. Any headings used herein are for convenience only and have no legal or limiting effect.

[0014] Specific details are provided in the following description to provide a thorough understanding of the embodiments. However, those skilled in the art will appreciate that the embodiments may be practiced without these specific details. For example, systems, processes, and other elements in the disclosed subject matter may be shown as components in block diagram form to avoid obscuring the embodiments in unnecessary detail. In other instances, well-known processes, structures, and techniques may be shown without unnecessary detail to avoid obscuring the embodiments. Additionally, like reference numbers and names in the various drawings refer to like elements. (System Overview)

[0015] FIG. 1 is a schematic block diagram 100 illustrating a training system 104 for video anomaly detection according to some embodiments of the present disclosure.

[0016] The training system 104 includes a memory 106, a first neural network 108, a second neural network 110, and a processor 112. The memory 106 includes at least one of RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical disk storage, magnetic cassette, magnetic tape, magnetic disk storage, or other storage medium usable to store desired information and accessible by the training system 104. The memory 106 may include non-transitory computer storage media in the form of volatile and / or non-volatile memory. The memory 106 may be removable, non-removable, or a combination thereof. Exemplary memory devices include solid state memory, hard drives, optical disk drives, and the like. The memory 106 stores instructions that are executed by the processor 112. Execution of the instructions by the processor 112 causes the training system 104 to perform a series of operations described below.

[0017] The training system 104 collects source domain data, such as a video of one or more digital images, from the source domain 102 to perform video anomaly detection in the target domain. In one example, the source domain 102 is any location, such as an airport, a shopping mall parking lot, a train station, etc. The videos in the source domain 102 correspond to at least one of task-unrelated or task-related videos, videos related to normal events, publicly available videos, and videos combining multiple images. The source domain data includes only normal events and does not include abnormal events. For example, a car driving on a road is considered a normal event, but when the car enters a pedestrian area it is considered an abnormal event, and a person running on a sports court is considered a normal event, but a person running outside a bank is considered an abnormal event.

[0018] In general, the target domain corresponds to anywhere where video anomaly detection is performed. As an example, the task corresponds to video anomaly detection. Furthermore, the training system 104 obtains a set of images of foreground objects present in the collected video from the source domain 102. The foreground objects correspond to at least one of a group of people and an individual. The source domain 102 may be different from the target domain. The source domain data is utilized as training data for training the first neural network 108 and the second neural network 110 for performing video anomaly detection.

[0019] Further, the first neural network 108 is trained to predict frames of one or more digital images in the video. Further, the second neural network 110 is trained to classify predicted frames of the digital images of the one or more digital images as normal or abnormal. For example, a predicted frame of the digital image of the one or more digital images having a corrupted portion with an image from the image set of a foreground object is classified as abnormal. Further, video anomaly detection is performed based on the training of the first neural network 108 and the second neural network 110. The anomalies in the video correspond to, but are not limited to, a fire, a fight, a collision of two or more objects, etc. The training of the first neural network 108 and the second neural network 110 is further described in Figures 2A, 2B, and 2C.

[0020] 2A illustrates an environment 200A for training a first neural network 108, according to some embodiments of the present disclosure. The zVAD system 104 is associated with a future frame prediction module 202. The first neural network 108 is trained using the future frame prediction module 202 to predict one or more frames of one or more digital images.

[0021]

number

[0022]

number

[0023]

number

[0024] Moreover, HMDB is a large collection of realistic videos from various sources, including movies and web videos. The dataset consists of 6,849 video clips from 51 action categories (e.g., "jumping", "kissing", "laughing"), with each category containing at least 101 clips. Finally, the HMDB dataset is utilized to learn human activities to perform the VAD task.

[0025]

number

[0026]

number

[0027]

number

[0028]

number

[0029]

number

[0030]

number

[0031]

number

[0032]

number

[0033]

number

[0034]

number

[0035]

number

[0036]

number

[0037]

number

[0038]

number

[0039] 6 is a flow chart illustrating a method 600 for performing video anomaly detection, according to various embodiments of the present disclosure. The method 600 is performed by the training system 104. The method 600 begins at step 602. Following step 602, the method 600 includes collecting a video of one or more digital images from the source domain 102 at step 604. The videos of the source domain 102 correspond to at least one of non-task related or task related videos, videos related to normal events, publicly available videos, and videos combining multiple images.

[0040] The method includes, in step 606, obtaining a set of images of a foreground object present in the video. The foreground object corresponds to at least one of a group of people and an individual. The method includes, in step 608, training a first neural network 108 to predict frames of one or more digital images in the video. The first neural network 108 is trained using a future frame prediction module 202 that predicts frames of the one or more digital images. The future frame prediction module 202 includes a generator 206 and a discriminator 212. The first neural network 108 is trained using one or a combination of a reconstruction loss and a discrimination loss. The reconstruction loss is calculated to train the generator 206. The discrimination loss is calculated to train the discriminator 212. The generator 206 includes an encoder 206A, a memory module 206B, and a decoder 206C. The future frame prediction module 202 is associated with an object recognition anomaly synthesis module 216. The object recognition anomaly synthesis module 216 generates pseudo anomalous frames and learns features from normal frames for anomalous frames.

[0041] The method includes, at step 610, training the second neural network 110 to classify predicted frames of the digital images of the one or more digital images as normal and to classify predicted frames of the digital images of the one or more digital images having corrupted portions with images from the set of images of the foreground object as anomalous. The second neural network is trained using a normality classifier module 214. The normality classifier module 214 generates a number of loss functions utilized to train the second neural network 110. The number of loss functions include a normality loss, a relative normality loss, an attention positive loss, and a relative attention positive loss. The normality classifier module 214 utilizes the predicted frames received from the future frame prediction module 202 and one or more pseudo anomalous frames generated by the object recognition anomaly synthesis module 216 to predict the probability of normality or anomaly in the video. The normality classifier module 214 is trained jointly using the predicted normal video frames and the pseudo anomalous video frames to detect anomalies in the video.

[0042] The method includes, at step 612, performing video anomaly detection based on the training of the first neural network 108 and the second neural network 110. The method ends at step 614.

[0043] FIG. 7 illustrates a use case for detecting anomalies using the training system 104, according to some example embodiments of the present disclosure. The zVAD system 104 can be integrated into a video surveillance system for detecting anomalies in various locations. The use case includes a parking lot scene 700. The scene 700 includes a video surveillance camera 702, a person A, and a person B. Person A and person B are seen fighting in the parking lot scene 700. A video of person A and person B fighting is captured in real time by the video surveillance camera 702. Furthermore, the video is detected as an anomaly based on the trained neural network of the training system 104. Furthermore, the training system 104 sends a video anomaly detection notification to a server, and appropriate action is taken immediately.

[0044] Many modifications and other embodiments of the inventions described herein will occur to one skilled in the art to which these inventions pertain having the benefit of the teachings presented in the foregoing description and the associated drawings. It is to be understood that the invention is not limited to the specific embodiments disclosed, and that modifications and other embodiments are intended to be included within the scope of the appended claims. Moreover, while the foregoing description and the associated drawings describe example embodiments in the context of examples of particular combinations of elements and / or features, it should be understood that different combinations of elements and / or features may be provided by alternative embodiments without departing from the scope of the appended claims. In this regard, combinations of elements and / or features other than those expressly described above are contemplated, for example, as may be set forth in some of the appended claims. Although specific terms have been employed herein, they are used in a generic and descriptive sense only and not for purposes of limitation.

Claims

1. 1. A computer-implemented method for video anomaly detection training, comprising: Obtaining a video of one or more digital images from a source domain; obtaining a set of images of foreground objects present in the video; training a first neural network to predict one or more frames of the one or more digital images in the video; and training a second neural network to classify the predicted one or more frames as at least one of normal and anomalous, wherein the predicted one or more frames of the one or more digital images without corruption are classified as normal and the predicted one or more frames of one or more of the one or more digital images having a corrupted portion of an image from the set of images of the foreground object are classified as anomalous, the method further comprising: performing the video anomaly detection based on training of the first neural network and the second neural network.

2. The method of claim 1 , wherein the foreground objects correspond to at least one of a group of people and an individual.

3. the first neural network is trained using a future frame prediction module to predict frames of the one or more digital images; The future frame prediction module includes a generator and a classifier; The method of claim 1 , wherein the generator includes an encoder, a memory module, and a decoder.

4. the first neural network is trained using one or a combination of a reconstruction loss and a discrimination loss; The reconstruction loss is computed to train the generator; The method of claim 3 , wherein the discrimination loss is calculated to train the discriminator.

5. The future frame prediction module is associated with an object recognition anomaly synthesis module; The method of claim 3 , wherein the object recognition anomaly synthesis module generates pseudo-anomalous frames and learns features from normal frames for anomalous frames.

6. The method of claim 5 , wherein the object recognition anomaly synthesis module is configured to generate one or more pseudo-anomalous frames and a ground truth mask.

7. The second neural network is trained using a normality classifier module; The method of claim 1 , wherein the normality classifier module is configured to generate a plurality of losses including a normality loss, a contrastive normality loss, a contrastive attention positive loss, and an attention positive loss.

8. 8. The method of claim 7, wherein the normality classifier module utilizes the predicted frames received from the future frame prediction module and the pseudo-anomalous frames generated by the object recognition anomaly synthesis module to predict a probability of normality or anomaly in the video.

9. 2. The method of claim 1 , wherein the first and second neural networks are jointly trained with the aid of a normality classifier module using predicted normal and pseudo abnormal video frames to detect anomalies in the video.

10. 2. The method of claim 1, wherein the videos of the source domain correspond to at least one of: non-task related videos, task related videos, general event related videos, publicly available videos, and videos combining multiple images.

11. 1. A training system comprising at least one processor and a memory storing instructions that, when executed by the at least one processor, cause the training system to: collecting a video of one or more digital images of a source domain; obtaining a set of images of foreground objects present in the video; training a first neural network to predict one or more frames of one or more digital images in the video based on a sequence of the one or more digital images in the video; training a second neural network to classify the predicted one or more frames as at least one of normal and anomalous, wherein the predicted one or more frames of the one or more digital images without corruption are classified as normal and the predicted one or more frames of the one or more digital images having a corrupted portion of an image from the set of images of the foreground object are classified as anomalous, the instructions further comprising: A training system for performing video anomaly detection based on training the first neural network and the second neural network.

12. the first neural network is trained using a future frame prediction module to predict frames of the one or more digital images; The future frame prediction module includes a generator and a classifier; The training system of claim 11 , wherein the generator includes an encoder, a memory module, and a decoder.

13. the first neural network is trained using one or a combination of a reconstruction loss and a discrimination loss; The reconstruction loss is computed to train the generator; The training system of claim 11 , wherein the discrimination loss is calculated to train the classifier.

14. The second neural network is trained using a normality classifier module; the normality classifier module generates a plurality of losses that are utilized to train the second neural network; The training system of claim 11 , wherein the plurality of losses include a normality loss, a relative normality loss, an attentional positive loss, and a relative attentional positive loss.

15. The normality classifier module is associated with an object recognition anomaly synthesis module; The training system of claim 14 , wherein the object recognition anomaly synthesis module generates pseudo-anomalous frames and learns features from normal frames for anomalous frames.

16. 15. The training system of claim 14, wherein the normality classifier module utilizes the predicted frames received from the future frame prediction module and the pseudo-anomalous frames generated by the object recognition anomaly synthesis module to predict a probability of normality or anomaly in the video.

17. 12. The training system of claim 11, wherein the videos of the source domain correspond to at least one of non-task related or task related videos, videos related to regular events, publicly available videos, and videos combining multiple images.

Images