Video Anomaly Detection Apparatus and Method using Relational Embedding

Abstract

The disclosed embodiment provides a video anomaly detection device and method that receives a video and divides it into multiple instances, performs neural network operations on each of the divided multiple instances to extract multiple temporal features, performs neural network operations on the extracted multiple temporal features to extract multiple motion features and multiple object features for each instance, estimates the relationship between the extracted multiple motion features and multiple object features to obtain multiple relationship features, obtains multiple temporal anomaly scores and multiple relationship anomaly scores from each of the multiple temporal features and multiple relationship features, and obtains multiple anomaly scores to determine whether each instance contains anomaly behavior by weighting the relationship anomaly scores corresponding to each temporal anomaly score, thereby determining an anomaly situation by considering not only temporal continuity but also the relationship with the surrounding environment, so as to accurately determine an anomaly situation and learn with high efficiency by utilizing all instances obtained from normal video for learning.

Description

Technology Field

[0001] The disclosed embodiments relate to a video anomaly detection device and method, and more specifically, to a video anomaly detection device and method using relational embedding of motion information and content information. Background Technology

[0002] Video anomaly detection is a technology that identifies frame segments containing abnormal situations (or abnormal situations), such as crime scenes or accident scenes, in input video. It replaces manual video monitoring and detection by humans and can play a role in rapid response to accidents and crime prevention.

[0003] When training anomaly detection techniques using supervised learning, a large volume of training videos is required, with truth values ​​labeled for each of the numerous frames acquired as training data; however, there are limitations in acquiring this data because anomalies occur very rarely compared to normal situations. Furthermore, the definition of anomalies is ambiguous, and classifying and acquiring data for all anomalies is not only impossible but also highly inefficient in terms of both time and cost.

[0004] Accordingly, unsupervised learning has been utilized to learn the distribution of normal patterns by leveraging a vast amount of normal situation data where truth values ​​are not labeled. However, this approach suffers from performance degradation due to the ambiguity of the normal situation category and the frequent detection of false positives, as it judges all non-normal situations as abnormal.

[0005] Meanwhile, recently, weakly supervised learning methods that use videos labeled with truth values ​​at the video level as training data are mainly being used.

[0006] In Multiple Instance Learning (MIL), a representative weakly supervised learning method, videos are divided into multiple instances. For each of the multiple instances obtained from anomaly videos labeled as anomaly situations and normal videos labeled as normal situations, learning is performed by assuming that the very small number of instances with the highest anomaly scores represent the instances where the anomaly occurred. As a result, inefficiency arises in which a large number of instances in anomaly videos are not utilized for training.

[0007] The reason MIL assumes here that only a very small number of instances are abnormal is that, since the weakly supervised learning method cannot pre-verify whether an abnormal situation is included in each instance, increasing the number of instances assumed to be abnormal could lead to normal instances being mistaken for abnormal instances during training.

[0008] However, since anomaly videos contain at least one instance according to the labeled truth value, in order to improve learning accuracy in MIL, training can be performed by assuming that only one instance among the multiple instances obtained from anomaly videos is an anomaly instance. In this case, although all instances obtained from normal situation videos labeled as normal can be used for training as they are identical to those labeled as normal, in practice, only one instance with the highest anomaly score from normal situation videos is selected and used for training in order to balance with the anomaly videos.

[0009] In other words, there is an efficiency issue in that all instances except for one instance among the multiple instances obtained from normal situation videos cannot be utilized for training.

[0010] In addition, abnormal and normal situations regarding specific behaviors are often relative rather than absolute. For example, in the case of a video involving a pedestrian, abnormal and normal situations can change depending on the surrounding environment, or background, where the pedestrian is walking. That is, if the place where the pedestrian is walking is a sidewalk, it should be judged as a normal situation, whereas if it is a roadway, it should be judged as an abnormal situation; similarly, when walking on a crosswalk, if the traffic light is green, it should be judged as a normal situation, whereas if it is red, it should be judged as an abnormal situation.

[0011] As such, although abnormal and normal situations should be determined differently based on the relative relationship between an object and its surrounding environment, there is a problem in that it is difficult to accurately identify abnormal situations because MIL is trained to determine the presence of an abnormal situation by simply comparing an estimated abnormal score with a fixed specific threshold value, thereby failing to consider the relative relationship between abnormal and normal situations. Prior art literature

[0012] Korean Published Patent No. 10-2022-0134180 (Published on Oct. 5, 2022) The problem to be solved

[0013] The disclosed embodiments aim to provide a video abnormal situation detection device and method capable of accurately determining an abnormal situation.

[0014] The disclosed embodiments aim to provide a video anomaly detection device and method capable of determining an anomaly by considering not only temporal continuity but also the relationship with the surrounding environment.

[0015] The disclosed embodiments aim to provide a video anomaly detection device and method in which all instances acquired from normal video are utilized for learning and learned with high efficiency. means of solving the problem

[0016] A video anomaly detection device according to an embodiment receives a video and divides it into multiple instances, performs neural network operations on each of the divided multiple instances to extract multiple temporal features, performs neural network operations on the extracted multiple temporal features to extract multiple motion features and multiple object features for each instance, estimates the relationship between the extracted multiple motion features and multiple object features to obtain multiple relationship features, obtains multiple temporal anomaly scores and multiple relationship anomaly scores from each of the multiple temporal features and the multiple relationship features, and obtains multiple anomaly scores for determining whether each instance contains an anomaly by weighting the relationship anomaly scores corresponding to each temporal anomaly score.

[0017] The processor can project the plurality of motion features and the plurality of object features into a virtual common embedding space to convert them into a graph, and perform neural network operations on the converted graph to estimate the relationship between the plurality of motion features and the corresponding object features to obtain the relationship features.

[0018] The processor can acquire features obtained in the process of detecting optical flow representing changes between temporal features by performing neural network operations on each of the plurality of temporal features as the plurality of motion features, and acquire features obtained in the process of detecting objects included in each temporal feature by performing neural network operations on each of the plurality of temporal features as the plurality of object features.

[0019] The processor can obtain a plurality of temporal anomaly scores by performing neural network operations on each of the plurality of temporal features, obtain a plurality of relationship anomaly scores by performing neural network operations on each of the plurality of relationship features, and obtain anomaly scores by adding relationship anomaly scores corresponding to each of the plurality of temporal anomaly scores.

[0020] The processor can extract multiple features by encoding each of multiple instances with a neural network operation, and obtain multiple temporal features by performing a neural network operation on each of the multiple features.

[0021] The processor can receive an abnormal situation video and a normal situation video, each labeled with a truth value, during training, and divide them into multiple abnormal situation instances and multiple normal situation instances, and calculate a temporal loss such that the difference between the top k abnormal situation temporal anomalies and the top k normal situation temporal anomalies among the multiple abnormal situation temporal anomalies and multiple normal situation temporal anomalies obtained from each of the multiple abnormal situation instances and the multiple normal situation instances is maximized, and backpropagate.

[0022] The processor receives an abnormal situation video and a normal situation video, each labeled with a truth value, during training, and separates them into multiple abnormal situation instances and multiple normal situation instances, and calculates a relationship loss to maximize the difference between the top k abnormal situation relationship scores and the top k normal situation relationship scores among the multiple abnormal situation relationship scores and the multiple normal situation relationship scores obtained from each of the multiple abnormal situation instances and the multiple normal situation instances, and backpropagates.

[0023] The processor receives an abnormal situation video and a normal situation video, each labeled with a truth value, during training, and separates them into multiple abnormal situation instances and multiple normal situation instances, and projects multiple abnormal situation relationship features and multiple normal situation relationship features obtained from each of the multiple abnormal situation instances and the multiple normal situation instances onto a virtual common embedding space, and calculates a distance loss and backpropagates based on the distance between the multiple normal situation relationship features projected into the common embedding space and the distance between the multiple abnormal situation relationship features and the multiple normal situation relationship features.

[0024] The processor can calculate the distance loss such that the distance between the plurality of normal situation relationship features is shortened, and the distance between the plurality of abnormal situation relationship features and some selected abnormal situation relationship features among the plurality of normal situation relationship features is widened.

[0025] The processor can calculate the distance loss by extracting the top k abnormal situation relationship scores and the top k normal situation relationship scores among the multiple abnormal situation relationship scores and the multiple normal situation relationship scores obtained from each of the multiple abnormal situation instances and the multiple normal situation instances, and by selecting the abnormal situation relationship feature and the normal situation relationship feature corresponding to each of the extracted top k abnormal situation relationship scores and the top k normal situation relationship scores.

[0026] A video anomaly detection method according to an embodiment comprises: a step of receiving a video and dividing it into a plurality of instances, and extracting a plurality of temporal features by performing neural network operations on each of the divided plurality of instances using an artificial neural network; a step of extracting a plurality of motion features and a plurality of object features for each instance by performing neural network operations on the extracted plurality of temporal features using an artificial neural network, and obtaining a plurality of relationship features by estimating the relationship between the extracted plurality of motion features and the plurality of object features; and a step of obtaining a plurality of temporal anomaly scores and a plurality of relationship anomaly scores from each of the plurality of temporal features and the plurality of relationship features, and obtaining a plurality of anomaly scores for determining whether an anomaly behavior is included in each instance by weighting the relationship anomaly scores corresponding to each temporal anomaly score. Effects of the invention

[0027] Accordingly, the video anomaly detection device and method according to the embodiment can accurately determine an anomaly by determining the anomaly by considering not only the temporal continuity of the video but also the relationship with the surrounding environment. In addition, all instances obtained from normal videos can be utilized for learning, enabling learning with high efficiency. Brief explanation of the drawing

[0028] FIG. 1 shows a configuration of a video abnormal situation detection device according to one embodiment, classified according to the operation performed. Figure 2 shows an example of the detailed configuration of the relationship estimation module of Figure 1. Figure 3 is a diagram illustrating the distance loss calculated in the loss calculation module of Figure 1. FIG. 4 illustrates a video abnormality detection method according to one embodiment. FIG. 5 is a diagram illustrating a computing environment including a computing device according to one embodiment. Specific details for implementing the invention

[0029] Hereinafter, a specific embodiment of one embodiment will be described with reference to the drawings. The following detailed description is provided to facilitate a comprehensive understanding of the methods, devices, and / or systems described herein. However, this is merely illustrative and the invention is not limited thereto.

[0030] In describing the embodiments, if it is determined that a detailed description of known technology related to the present invention might unnecessarily obscure the essence of the embodiment, such detailed description will be omitted. Furthermore, the terms described below are defined with consideration of their functions in the present invention, and these may vary depending on the intentions or practices of the user or operator. Therefore, such definitions should be based on the content throughout this specification. Terms used in the detailed description are intended merely to describe the embodiments and should not be limiting. Unless explicitly stated otherwise, expressions in the singular form include the meaning of the plural form. In this description, expressions such as "include" or "comprise" are intended to refer to certain characteristics, numbers, steps, actions, elements, parts thereof, or combinations thereof, and should not be interpreted to exclude the existence or possibility of one or more other characteristics, numbers, steps, actions, elements, parts thereof, or combinations thereof other than those described. Additionally, terms such as "...part," "...unit," "module," and "block" described in the specification refer to a unit that processes at least one function or operation, and this may be implemented in hardware, software, or a combination of hardware and software.

[0031] FIG. 1 shows a configuration of a video abnormal situation detection device according to one embodiment, classified according to the operation performed, and FIG. 2 shows an example of a detailed configuration of the relationship estimation module of FIG. 1.

[0032] Referring to FIGS. 1 and 2, another video anomaly detection device in the embodiment may include a video input module (11), a temporal feature extraction module (12), a relationship estimation module (13), an anomaly score acquisition module (14), and a loss calculation module (20).

[0033] The video input module (11) receives a training video prepared in advance for weakly supervised learning during training. Here, the training video includes both anomaly video and normal video, and each of the anomaly video and normal video is a video in which it is checked in advance whether an anomaly is included in the entire frame interval rather than on an individual frame basis and labeled with a ground truth value. The anomaly video is a video in which an anomaly is included in at least one frame and the anomaly is labeled with a ground truth value, and the normal video is a video in which an anomaly is not included in any frames and the normal is labeled with a ground truth value.

[0034] The abnormal situation video and the normal situation video may be input sequentially into the video input module (11), but in some cases, they may be input together.

[0035] When an abnormal situation video and a normal situation video are input, the video input module (11) separates each of the input abnormal video and normal video into multiple instances (instances or clips). Generally, in a weakly supervised learning technique, a video composed of multiple frames is considered as a single bag (B), and multiple frames of each bag (B) are grouped into multiple instances, and whether an abnormal situation is included in each separated instance is determined.

[0036] The video anomaly detection device of the embodiment also divides the test video input during a test operation into multiple instances, just like the conventional one, and identifies the instance containing an anomaly among the divided multiple instances.

[0037] Accordingly, the video input module (11) distinguishes between abnormal videos and normal videos, each applied during training, into multiple instances. At this time, the abnormal video is a Positive Bag (B a It is called ), and the normal video is the Negative Bag (B n It is called ). And below, for the convenience of explanation, positive back positive back (B a Multiple instances obtained from ) are called abnormal situation instances, and negative back(B n Multiple instances obtained from ) are called normal situation instances.

[0038] In this case, multiple instances of normal situation acquired from a normal situation video labeled as normal because no abnormal situation is included are equivalent to a normal situation being labeled. In contrast, multiple instances of abnormal situation acquired from an abnormal situation video labeled as abnormal situation contain an abnormal situation in at least one instance, but not all instances contain an abnormal situation.

[0039] The temporal feature extraction module (12) extracts temporal features that consider temporal correlations from multiple normal situation instances and multiple abnormal situation instances. Basically, abnormal situations or normal situations are caused by the actions or behaviors of an object. And these actions or behaviors do not occur in an extremely short moment, but rather occur with continuity in which temporal context exists. The temporal feature extraction module (12) extracts temporal features to detect abnormal situations by considering this temporal continuity.

[0040] The temporal feature extraction module (12) may include an encoder (15) and a temporal correlation module (16), each implemented as an artificial neural network. The encoder (15) performs neural network operations on each of a plurality of abnormal situation instances and a plurality of normal situation instances to extract a plurality of abnormal situation features and a plurality of normal situation features. Here, the encoder (15) may use an artificial neural network that has been pre-trained and made public to extract features from an image.

[0041] The temporal correlation module (16) performs neural network operations on multiple abnormal situation features extracted from the encoder (15) and performs neural network operations on multiple normal situation features to extract multiple temporal features that reflect the temporal dependencies between multiple abnormal situation features and the temporal dependencies between multiple normal situation features. The temporal correlation module (16) may be implemented, for example, as a Recurrent Neural Network (RNN), Long Short Term Memory (LSTM), or Bi-LSTM (Bidirectional LSTM) in which features can propagate along the time axis, but is not limited thereto. The temporal correlation module (16) may also extract temporal features by varying the kernel size of the artificial neural network to vary the interval in which features propagate. The temporal correlation module (16) may also obtain multiple abnormal situation temporal features and multiple normal situation temporal features for each of the abnormal situation video and the normal situation video.

[0042] That is, the temporal correlation module (16) can obtain multiple abnormal situation temporal features by reflecting the features of an abnormal situation instance from a previous or subsequent time period into the abnormal situation features extracted from each abnormal situation instance, and obtain multiple normal situation temporal features by reflecting the features of a normal situation instance from a previous or subsequent time period into the normal situation features extracted from each normal situation instance.

[0043] Here, for the sake of convenience of understanding, the artificial neural network implementing the encoder (15) and the temporal correlation module (16) has been described separately, but the encoder (15) and the temporal correlation module (16) may be implemented as an integrated artificial neural network.

[0044] Meanwhile, the relationship estimation module (13) receives multiple abnormal situation temporal features and multiple normal situation temporal features obtained from the temporal feature extraction module (12), and extracts relationship features that estimate the relationship between motion and surrounding environment, particularly the background, in each instance.

[0045] As illustrated in FIG. 2, the relationship estimation module (13) may include a relationship modeling module (31) and a relationship feature extraction module (35).

[0046] The relationship modeling module (31) extracts motion features representing object movement in each instance and object features for identifying objects from each of the multiple abnormal situation temporal features and multiple normal situation temporal features, and converts the extracted motion features and object features into a graph so that the relationship between the extracted motion features and object features can be easily verified.

[0047] The relationship modeling module (31) may include a motion feature extraction module (32), an object feature extraction module (33), and a graph transformation module (34). The motion feature extraction module (32) extracts motion features by performing neural network operations on each of a plurality of abnormal situation temporal features and a plurality of normal situation temporal features. The motion feature extraction module (32) may be implemented as an artificial neural network that estimates optical flow for each input feature, and an artificial neural network that has been previously trained and known to estimate optical flow may be used. Specifically, the motion feature extraction module (32) may obtain a plurality of optical flows representing the positional change of corresponding pixels between consecutive abnormal situation instances among a plurality of abnormal situation instances and between consecutive normal situation instances among a plurality of normal situation instances by estimating them through neural network operations.

[0048] However, in the embodiment, the motion feature extraction module (32) can obtain features obtained during the process of estimating a number of abnormal situation optical flows and a number of normal situation optical flows through neural network computation as motion features. That is, features obtained from a layer located before the final layer among a number of layers of an artificial neural network that estimates optical flows can be extracted and obtained as motion features.

[0049] Additionally, the motion feature extraction module (32) may calculate the average size for each of the estimated multiple optical flows and select optical flows in which the calculated average size is greater than or equal to a specified reference size, thereby extracting only the motion features corresponding to the selected optical flows. This may also extract only the motion features for instances containing large movements among multiple abnormal situation instances and multiple normal situation instances. This is intended to allow the selection of instances where normal or abnormal behavior occurs, i.e., sections in the video where large movements are captured, and to establish relationships by extracting only the motion features with large movements.

[0050] Meanwhile, the object feature extraction module (33) extracts object features by performing neural network operations on each of the multiple abnormal situation temporal features and the multiple normal situation temporal features. The object feature extraction module (33) can be implemented as an artificial neural network that detects abnormal situation objects and normal situation objects in each of the input multiple abnormal situation temporal features and multiple normal situation temporal features, and one of several object detection artificial neural networks that have been trained and publicized in advance can be used. In addition, the object feature extraction module (33) can also acquire features obtained during the process of detecting multiple abnormal situation objects and multiple normal situation objects as object features.

[0051] In addition, the object feature extraction module (33) can extract object features for the remaining instances excluding the instances from which motion features were extracted, when the motion feature extraction module (32) selects optical flows larger than a reference size and extracts only motion features for some instances.

[0052] As mentioned above, since abnormal or normal situations can be defined by the relationship between the object's behavior and the surrounding environment, the object feature extraction module (33) is designed to extract object features by focusing on various backgrounds that correspond to the surrounding environment.

[0053] The graph conversion module (34) converts the multiple abnormal situation motion features and multiple normal situation motion features extracted by the motion feature extraction module (32) into a graph to facilitate the extraction of relationships between the multiple abnormal situation object features and multiple normal situation object features extracted by the object feature extraction module (33). The graph conversion module (34) can convert each of the multiple motion features and multiple object features into a relationship feature graph by projecting them onto a virtual common embedding space.

[0054] The relationship feature extraction module (35) performs neural network operations on a graph in which motion features and object features are projected onto an embedding space and transformed to estimate the relationship between multiple motion features and multiple object features and extract relationship features. The relationship feature extraction module (35) can extract abnormal relationship features between abnormal motion features and abnormal object features, and normal relationship features between normal motion features and normal object features.

[0055] Since motion features are extracted by paying attention to the movement, or behavior, of an object in each instance, and object features are extracted by paying attention to the object, particularly its surrounding environment, the relationship features extracted by combining motion features and object features in a common embedding space can be said to represent the relationship between the object's behavior and the surrounding environment.

[0056] The abnormal score acquisition module (14) determines the abnormal situation level based on multiple temporal features acquired from the temporal feature extraction module (12) and multiple relationship features acquired from the relationship estimation module (13) to acquire an abnormal score.

[0057] The abnormal score acquisition module (14) may include a temporal abnormal score determination module (17), a relationship score determination module (18), and an abnormal score determination module (19).

[0058] The temporal anomaly score determination module (17) determines a temporal anomaly score, which is a score obtained by analyzing an anomaly situation included in each instance from a temporal perspective, based on multiple temporal features extracted from each of multiple instances.

[0059] The temporal anomaly score determination module (17) performs neural network operations on each of the multiple abnormal situation temporal features and each of the multiple normal situation temporal features obtained from the temporal feature extraction module (12) to estimate and obtain an anomaly score for each abnormal situation temporal feature and each normal situation temporal feature. Here, the anomaly score can be estimated in the form of a probability value that an abnormal situation is included in the abnormal instance corresponding to each abnormal situation temporal feature and the normal instance corresponding to each normal situation temporal feature, but can be set in various ways depending on the learning method.

[0060] The temporal anomaly score determination module (17) performs neural network operations on each of the multiple abnormal situation temporal features and each of the multiple normal situation temporal features obtained from the temporal feature extraction module (12) to estimate and obtain a temporal anomaly score, which is an anomaly score for each abnormal situation temporal feature and each normal situation temporal feature. Here, the temporal anomaly score can be estimated in the form of a probability value that an abnormal situation is included in the abnormal instance corresponding to each temporal feature and the normal instance corresponding to each normal situation temporal feature, but can be set in various ways depending on the learning method.

[0061] The relationship anomaly score determination module (18) performs neural network operations on each of the multiple abnormal situation relationship features and each of the multiple normal situation relationship features obtained from the relationship estimation module (13) to estimate and obtain a relationship anomaly score, which is an anomaly score for each abnormal situation relationship feature and each normal situation relationship feature. The relationship anomaly score can also be estimated in the form of a probability value indicating that an abnormal situation is included in an abnormal instance corresponding to a temporal feature and a normal instance corresponding to each normal situation temporal feature, based on the relationship between the object's behavior and the surrounding environment.

[0062] The temporal anomaly score determination module (17) and the relationship anomaly score determination module (18) can be implemented as an artificial neural network, and as an example, can be implemented as a fully connected layer.

[0063] The anomaly score determination module (19) can obtain an anomaly score by weighting each of the multiple temporal anomaly scores estimated by the temporal anomaly score determination module (17) with the multiple relationship anomaly scores estimated by the relationship anomaly score determination module (18). The anomaly score determination module (19) can obtain an anomaly score by summing the temporal anomaly scores and relationship anomaly scores extracted from the same instance among the multiple temporal anomaly scores and multiple relationship anomaly scores. That is, an anomaly score is obtained in which the anomaly situation estimated from the temporal characteristics of each instance and the anomaly situation estimated from the relational characteristics are considered together.

[0064] Additionally, the abnormal score determination module (19) may determine whether each instance contains an abnormal situation based on the acquired abnormal score.

[0065] The loss calculation module (20) receives a plurality of temporal anomaly scores from the temporal anomaly score determination module (17) and a plurality of relationship anomaly scores from the relationship anomaly score determination module (18), calculates the loss, and backpropagates to train the artificial neural networks of the temporal feature extraction module (12), the relationship estimation module (13), and the anomaly score acquisition module (14).

[0066] The loss calculation module (20) receives a plurality of abnormal situation temporal abnormal scores and a plurality of normal situation temporal abnormal scores from the temporal abnormal score determination module (17), and extracts the top k specified abnormal situation temporal abnormal scores and normal situation temporal abnormal scores from each of the received plurality of abnormal situation temporal abnormal scores and a plurality of normal situation temporal abnormal scores. Then, the temporal loss (L) is calculated such that the difference between the extracted top k (e.g., 1) abnormal situation temporal abnormal scores and the top k normal situation temporal abnormal scores is maximized. tem ) rank loss(l(B a , B n It can be calculated in the format ))

[0067] Additionally, the loss calculation module (20) receives a plurality of abnormal situation relationship abnormal scores and a plurality of normal situation relationship abnormal scores from the relationship abnormal score determination module (17), extracts the top k designated abnormal situation temporal abnormal scores and normal situation temporal abnormal scores from each of the received plurality of abnormal situation relationship abnormal scores and the plurality of normal situation relationship abnormal scores, and calculates a relationship loss (L) such that the difference between the extracted top k abnormal situation temporal abnormal scores and the top k normal situation temporal abnormal scores is maximized. rel ) can be calculated in the form of rank loss.

[0068] Here, rank loss(l(B a , B n )) can be calculated as Equation 1 when k is 1 and one or more scores are extracted.

[0069]

[0070] Here l(B a , B n ) is a positive back(B a ) and negative back(B n Represents the rank loss for ), and V a i and V n i are each positive back(B a ) and negative back(B n Represents the i-th instance extracted from ), and f(V a i ) and f(V n i ) are each instances (V a i , V n i Indicates the anomaly score (temporal anomaly score or relationship anomaly score) for ).

[0071] In addition, in the embodiment, the loss calculation module (20) calculates the loss based on the relative similarity between a normal situation and an abnormal situation from the process in which the relationship estimation module (13) extracts a plurality of motion features and a plurality of object features, and projects each of the extracted plurality of motion features and a plurality of object features onto a virtual common embedding space to convert them into a relationship feature graph, thereby extracting relationship features, the loss based on the distance loss (L) in the embedding space. dist It can be further calculated in the format ).

[0072] Figure 3 is a diagram illustrating the distance loss calculated in the loss calculation module of Figure 1.

[0073] As shown in FIG. 3, the loss calculation module (20) has a negative back (B n Multiple normal condition instances (V) obtained from ) n Multiple steady-state relationship features extracted from each of ) (f nWhile ) are placed close to each other in the embedding space, similar to rank loss, k anomaly relationship features (f) having the top k anomaly relationship anomaly scores and the top k normal relationship anomaly scores a ) and k steady-state relationship features (f n The similarity between ) is distance loss (L) such that a difference of more than a certain level occurs. dist ) can be calculated.

[0074] The loss calculation module (20) first calculates a number of normal situation instances (V n Multiple steady-state relationship features extracted from ) (f n Accumulating the cosine similarity between ) to obtain the steady-state relationship similarity (CS N Calculate ) according to mathematical formula 2.

[0075]

[0076] Here, T is the negative back(B n Normal situation instance (V) included in ) n Represents the number of ), and i, j represent the instance indices.

[0077] And k anomaly relationship features (f) having the top k anomaly relationship anomaly scores and the top k normal relationship anomaly scores a topk ) and k steady-state relationship features (f n topk The cosine similarity between ) is calculated according to Equation 3 to obtain the upper relationship similarity (CS Top Acquires ).

[0078]

[0079] Although some normal situations in reality may share characteristics with abnormal situations, abnormal and normal situations must nevertheless be clearly distinguishable; therefore, the characteristics of the normal situation relationship (f n Normal Situation Relationship Similarity (CS), which is the similarity between ) NCharacteristics of the relationship between ) and abnormal situations (f a ) and normal situation relationship characteristics (f n Features with upper or higher scores in ) (f a , f n Parent relationship similarity (CS), which is the similarity between ) Top There must be a clearly distinguishable difference between them. That is, as shown in Equation 4, the upper relationship similarity (CS) with added margin Top ) normal situation relationship similarity (CS N It must be smaller than ).

[0080]

[0081] Accordingly, distance loss (L dist ) can be calculated using mathematical formula 5.

[0082]

[0083] The loss calculation module (20) calculates the time loss (L) calculated as rank loss as in mathematical formula 1. tem ) and relationship loss (L rel ) and distance loss (L dist Weighted sum of ) to obtain total loss (L total ) can be calculated and backpropagated according to mathematical formula 6.

[0084]

[0085] Here, λ a and λ b are the respective loss weights.

[0086] Consequently, the video anomaly detection device according to the embodiment can receive a training video in which the inclusion of an anomaly or a normal situation at the video level is labeled as a truth value and be trained with weak supervision. Since it is trained to detect anomalies by reflecting not only the temporal features of multiple instances obtained from the received video, but also the relationship between motion features according to movement and object features regarding the surrounding environment, it can detect anomalies very accurately. Furthermore, by distinguishing between normal situation relationship features and anomaly situation relationship features extracted from multiple normal situation instances based on similarity, multiple normal situation instances obtained from normal situation videos can be utilized for training, thereby enabling effective training.

[0087] In the illustrated embodiments, each component may have different functions and capabilities other than those described below, and may include additional components other than those not described below. Additionally, in one embodiment, each component may be implemented using one or more physically separated devices, or by one or more processors or a combination of one or more processors and software, and may not be clearly distinguished in specific operation as in the illustrated examples.

[0088] And the video anomaly detection device illustrated in FIG. 1 may be implemented in a logic circuit by hardware, firmware, software, or a combination thereof, or may be implemented using a general-purpose or specific-purpose computer. The device may be implemented using a hardwired device, a field programmable gate array (FPGA), an application-specific integrated circuit (ASIC), etc. Additionally, the device may be implemented as a system-on-chip (SoC) including one or more processors and controllers.

[0089] In addition, the video anomaly detection device may be installed in a computing device or server equipped with hardware elements in the form of software, hardware, or a combination thereof. A computing device or server may refer to various devices that include, in whole or in part, communication devices such as communication modems for communicating with various devices or wired / wireless communication networks, memory for storing data for executing programs, and microprocessors for executing programs to perform calculations and commands.

[0090] FIG. 4 illustrates a video abnormality detection method according to one embodiment.

[0091] Referring to FIGS. 1 to 3, the video abnormal situation detection method of FIG. 4 can be divided into a weakly supervised learning stage and a test stage.

[0092] In the weak-supervised learning stage, a training video is first received as input (41). Here, the training video may include a normal situation video and an abnormal situation video, each labeled as normal and abnormal as truth values. The normal situation video and the abnormal situation video may be applied sequentially or together.

[0093] When normal video and abnormal video are input, each input normal video and abnormal video is divided into multiple normal video instances and multiple abnormal video instances, and each of the divided multiple normal video instances and multiple abnormal video instances is encoded by neural network operation using an artificial neural network to extract multiple normal video features and multiple abnormal video features representing each of the multiple normal video instances and multiple abnormal video instances (42).

[0094] Then, by reflecting the temporal correlation between each of the multiple normal situation features and multiple abnormal situation features, multiple normal situation temporal features and multiple abnormal situation temporal features are obtained (43).

[0095] Subsequently, multiple motion features are extracted by estimating optical flow through neural network operations using an artificial neural network for each of the multiple normal situation temporal features and multiple abnormal situation temporal features obtained thereafter, and multiple object features are extracted by detecting objects through neural network operations using an artificial neural network for each of the multiple normal situation temporal features and multiple abnormal situation temporal features obtained thereafter (44). Here, multiple motion features and multiple object features are features extracted during the process of optical flow and object detection using an artificial neural network. Also, multiple motion features and multiple object features can be classified into abnormal situation motion features and normal situation motion features and abnormal situation object features and normal situation object features depending on the corresponding instance.

[0096] Meanwhile, a number of abnormal situation motion features, a number of normal situation motion features, a number of abnormal situation object features, and a number of normal situation object features are projected into a virtual embedding space and converted into a graph (45).

[0097] Then, multiple abnormal situation relationship features representing the relationship between each of the multiple abnormal situation motion features and multiple abnormal situation object features converted into a graph, and multiple normal situation relationship features representing the relationship between each of the multiple normal situation motion features and multiple normal situation object features are extracted by performing neural network operations on an artificial neural network (46).

[0098] Meanwhile, neural network operations using an artificial neural network are performed on the obtained multiple normal situation temporal features and multiple abnormal situation temporal features to obtain multiple normal situation temporal abnormal scores and multiple abnormal situation temporal abnormal scores (47).

[0099] In addition, neural network operations are performed using an artificial neural network on multiple abnormal situation relationship features and multiple normal situation relationship features to obtain multiple abnormal situation relationship abnormal scores and multiple normal situation relationship abnormal scores (48).

[0100] Accordingly, a loss is calculated and backpropagated based on multiple normal situation temporal anomaly scores, multiple abnormal situation temporal anomaly scores, multiple abnormal situation relationship anomaly scores, and multiple normal situation relationship anomaly scores (49). At this time, a temporal loss (L) based on multiple normal situation temporal anomaly scores and multiple abnormal situation temporal anomaly scores is calculated. tem Calculate ) and based on multiple abnormal situation relationship anomaly scores and multiple normal situation relationship anomaly scores, the relationship loss (L rel ) can be calculated. Time loss (L tem ) and relationship loss (L rel ) can be calculated as a rank loss that maximizes the difference between the top k anomaly scores in each of the temporal anomaly scores and relationship anomaly scores extracted from normal and abnormal instances.

[0101] Additionally, the loss based on the relative similarity between normal and abnormal situations from the process of extracting relationship features by extracting multiple motion features and multiple object features, and transforming each of the extracted multiple motion features and multiple object features into a relationship feature graph by projecting them onto a virtual common embedding space, is the distance loss (L) in the embedding space. dist It can be calculated as ).

[0102] Time loss (L tem ) and relationship loss (L rel ) and distance loss (L dist If ) is calculated (L totalArtificial neural networks included in the abnormal situation detection device can be trained by calculating and backpropagating the value. At this time, some of the artificial neural networks included in the abnormal situation detection device may be pre-trained depending on the application, but they can be adjusted to suit the abnormal situation detection device of the embodiment through additional training in which the calculated loss is backpropagated.

[0103] Afterward, it is determined whether the training is terminated (80). At this time, the training is terminated when the training has been repeated a certain number of times or when the total loss (L) is calculated. total If this loss is below the threshold, it may be terminated.

[0104] Once the learning is finished and the weak map learning phase is complete, the test phase can be performed.

[0105] In the test phase, a test video with unlabeled truth values ​​is input (51). When the test video is input, the input test video is divided into multiple test instances. Then, for the multiple test instances, a temporal anomaly score and a relationship anomaly score are obtained according to the method already learned in the weakly supervised learning phase, and the obtained temporal anomaly score and relationship anomaly score are summed to obtain an anomaly score (52). Based on the obtained anomaly score, an anomaly instance among the input multiple test instances is identified (53).

[0106] Although FIG. 4 describes each process as being executed sequentially, this is merely an illustrative description, and a person skilled in the art can apply various modifications and variations by changing the order described in FIG. 4, executing one or more processes in parallel, or adding other processes, within the scope of not departing from the essential characteristics of the embodiment of the present invention.

[0107] FIG. 5 is a diagram illustrating a computing environment including a computing device according to one embodiment.

[0108] In the illustrated embodiments, each component may have different functions and capabilities in addition to those described below, and may include additional components in addition to those not described below. The illustrated computing environment (60) may include a computing device (61) to perform the video anomaly detection method illustrated in FIG. 4. In one embodiment, the computing device (61) may be one or more components included in the video anomaly detection device of FIG. 1.

[0109] The computing device (61) includes at least one processor (62), a computer-readable storage medium (63), and a communication bus (65). The processor (62) may enable the computing device (61) to operate according to the exemplary embodiment described above. For example, the processor (62) may execute one or more programs (64) stored in the computer-readable storage medium (63). The one or more programs (64) may include one or more computer-executable instructions, and the computer-executable instructions may be configured to enable the computing device (61) to perform operations according to the exemplary embodiment when executed by the processor (62).

[0110] The communication bus (65) interconnects various other components of the computing device (61), including the processor (62) and the computer-readable storage medium (63).

[0111] The computing device (61) may also include one or more input / output interfaces (66) and one or more communication interfaces (67) that provide interfaces for one or more input / output devices (68). The input / output interfaces (66) and the communication interfaces (67) are connected to a communication bus (65). The input / output devices (68) may be connected to other components of the computing device (61) through the input / output interfaces (66). An exemplary input / output device (68) may include input devices such as a pointing device (such as a mouse or trackpad), a keyboard, a touch input device (such as a touchpad or touchscreen), a voice or sound input device, various types of sensor devices and / or imaging devices, and / or output devices such as a display device, a printer, a speaker and / or a network card. An exemplary input / output device (68) may be included inside the computing device (61) as a component constituting the computing device (61), or it may be connected to the computing device (61) as a separate device distinct from the computing device (61).

[0112] Although the present invention has been described in detail above through representative embodiments, those skilled in the art will understand that various modifications and equivalent alternative embodiments are possible therefrom. Accordingly, the true technical scope of protection of the present invention should be determined by the technical spirit of the appended claims. Explanation of the symbols

[0113] 11: Video Input Module 12: Temporal Feature Extraction Module 13: Relationship Estimation Module 14: Anomaly Score Acquisition Module 15: Encoder 16: Temporal Correlation Module 17: Temporal Anomaly Score Determination Module 18: Relationship Score Determination Module 19: Anomaly Score Determination Module 20: Loss Calculation Module

Claims

Claim 1 One or more processors; A device comprising a memory for storing one or more programs executed by the one or more processors, wherein the processor receives a video and divides it into a plurality of instances, performs neural network operations on each of the divided plurality of instances using an artificial neural network to extract a plurality of temporal features, performs neural network operations on the extracted plurality of temporal features using an artificial neural network to extract a plurality of motion features and a plurality of object features for each instance, estimates the relationship between the extracted plurality of motion features and a plurality of object features to obtain a plurality of relationship features, obtains a plurality of temporal anomaly scores and a plurality of relationship anomaly scores from each of the plurality of temporal features and the plurality of relationship features, and obtains a plurality of anomaly scores for determining whether an abnormal behavior is included in each instance by weighting the relationship anomaly scores corresponding to each temporal anomaly score, wherein the processor receives an abnormal situation video and a normal situation video in which an abnormal situation or a normal situation is labeled as a truth value during learning, and divides them into a plurality of abnormal situation instances and a plurality of normal situation instances, respectively, and projects a plurality of abnormal situation relationship features and a plurality of normal situation relationship features obtained from each of the plurality of abnormal situation instances and the plurality of normal situation instances onto a virtual common embedding space, and the common embedding A video anomaly detection device that calculates distance loss and backpropagates based on the distance between the plurality of normal situation relationship features projected into space and the distance between the plurality of abnormal situation relationship features and the plurality of normal situation relationship features. Claim 2 A video anomaly detection device according to claim 1, wherein the processor projects the plurality of motion features and the plurality of object features into a virtual common embedding space to convert them into a graph, and performs neural network operations on the converted graph to estimate the relationship between the plurality of motion features and the corresponding object features to obtain the relationship features. Claim 3 A video anomaly detection device according to claim 1, wherein the processor obtains features obtained in the process of detecting optical flow representing a change between temporal features by performing neural network operations on each of the plurality of temporal features as the plurality of motion features, and obtains features obtained in the process of detecting objects included in each temporal feature by performing neural network operations on each of the plurality of temporal features as the plurality of object features. Claim 4 A video anomaly detection device according to claim 1, wherein the processor obtains a plurality of temporal anomaly scores by performing neural network operations on each of the plurality of temporal features, obtains a plurality of relationship anomaly scores by performing neural network operations on each of the plurality of relationship features, and obtains an anomaly score by adding relationship anomaly scores corresponding to each of the plurality of temporal anomaly scores. Claim 5 A video anomaly detection device according to claim 1, wherein the processor encodes each of a plurality of instances into a neural network operation to extract a plurality of features, and performs a neural network operation on each of the plurality of features to obtain the plurality of temporal features. Claim 6 delete Claim 7 delete Claim 8 delete Claim 9 A video anomaly detection device according to claim 1, wherein the processor calculates the distance loss such that the distance between the plurality of normal situation relationship features is shortened, and the distance between the plurality of abnormal situation relationship features and some abnormal situation relationship features selected among the plurality of normal situation relationship features and the normal situation relationship features is widened. Claim 10 A video anomaly detection device according to claim 9, wherein the processor extracts the top k anomaly relationship anomaly scores and the top k normal situation relationship anomaly scores among the multiple anomaly relationship anomaly scores and the multiple normal situation relationship anomaly scores obtained from each of the multiple anomaly situation instances and the multiple normal situation instances, and calculates the distance loss by selecting an anomaly relationship feature and a normal situation relationship feature corresponding to each of the extracted top k anomaly relationship anomaly scores and the top k normal situation relationship anomaly scores. Claim 11 A method performed by a computing device having one or more processors and a memory storing one or more programs executed by said one or more processors, comprising: receiving a video and dividing it into a plurality of instances, and extracting a plurality of temporal features by performing neural network operations on each of the divided plurality of instances using an artificial neural network; and performing neural network operations on the extracted plurality of temporal features using an artificial neural network to extract a plurality of motion features and a plurality of object features for each instance, and estimating the relationship between the extracted plurality of motion features and the plurality of object features to obtain a plurality of relationship features. A video anomaly detection method comprising the step of obtaining a plurality of temporal anomaly scores and a plurality of relationship anomaly scores from each of the plurality of temporal features and the plurality of relationship features, and obtaining a plurality of anomaly scores for determining whether anomaly behavior is included in each instance by weighting the relationship anomaly scores corresponding to each temporal anomaly score, wherein in the learning step of training the artificial neural network, an anomaly video and a normal video, each labeled as an anomaly or a normal situation with a truth value, are received and classified into a plurality of anomaly instances and a plurality of normal instances, respectively; a plurality of anomaly relationship features and a plurality of normal situation relationship features obtained from each of the plurality of anomaly instances and the plurality of normal instances are projected onto a virtual common embedding space; and a distance loss is calculated and backpropagated based on the distance between the plurality of normal situation relationship features projected into the common embedding space and the distance between the plurality of anomaly relationship features and the plurality of normal situation relationship features. Claim 12 A video anomaly detection method according to claim 11, wherein the step of acquiring the relationship feature comprises projecting the plurality of motion features and the plurality of object features into a virtual common embedding space to convert them into a graph, and performing neural network operations on the converted graph to estimate the relationship between the plurality of motion features and the corresponding object features to acquire the relationship feature. Claim 13 A video anomaly detection method according to claim 11, wherein the step of acquiring the relationship feature involves acquiring the feature obtained in the process of detecting an optical flow representing a change between temporal features by performing neural network operations on each of the plurality of temporal features as the plurality of motion features, and acquiring the feature obtained in the process of detecting an object included in each temporal feature by performing neural network operations on each of the plurality of temporal features as the plurality of object features. Claim 14 A video anomaly detection method according to claim 11, wherein the step of obtaining the anomaly score involves performing neural network operations on each of the plurality of temporal features to obtain the plurality of temporal anomaly scores, performing neural network operations on each of the plurality of relational features to obtain the plurality of relational anomaly scores, and adding the relational anomaly scores corresponding to each of the plurality of temporal anomaly scores to obtain the anomaly score. Claim 15 In claim 11, the step of extracting temporal features comprises encoding each of a plurality of instances into a neural network operation to extract a plurality of features, and performing a neural network operation on each of the plurality of features to obtain the plurality of temporal features, in a video anomaly detection method. Claim 16 delete Claim 17 delete Claim 18 delete Claim 19 A video anomaly detection method according to claim 11, wherein the learning step calculates the distance loss such that the distance between the plurality of normal situation relationship features is shortened, and the distance between the plurality of abnormal situation relationship features and some abnormal situation relationship features selected among the plurality of normal situation relationship features and the normal situation relationship features is widened. Claim 20 A video anomaly detection method according to claim 19, wherein the learning step extracts the top k anomaly relationship anomaly scores and the top k normal situation relationship anomaly scores among the multiple anomaly relationship anomaly scores and the multiple normal situation relationship anomaly scores obtained from each of the multiple anomaly situation instances and the multiple normal situation instances, and calculates the distance loss by selecting anomaly relationship features and normal situation relationship features corresponding to each of the extracted top k anomaly relationship anomaly scores and the top k normal situation relationship anomaly scores.

Images