Video anomaly detection apparatus based on cross-learning of segment-level and bag-level features, and learning apparatus and method thereof

Abstract

One embodiment of the present invention provides a video anomaly detection device based on cross-learning of segment unit features and back unit features, comprising: a segment division unit that divides an input video image into segment units composed of a plurality of temporally consecutive frames; a feature extraction unit that generates a segment feature vector representing the spatiotemporal features of the segment from the divided segment; and an anomaly detection unit that inputs the segment feature vector into a segment neural network trained to make a judgment considering the temporal context to determine whether there is an anomaly, wherein the segment neural network is trained through cross-learning with a back neural network configured to learn the contextual relationships of a plurality of temporally consecutive segments.

Description

Technology Field

[0001] The present invention relates to a technology for detecting abnormal behavior in video images, and more specifically, to a video anomaly detection device based on cross-learning of segment-unit features and hundreds-unit features, capable of detecting abnormal behavior without labeling through cross-learning of segment-unit features and hundreds-unit features, and to the learning device and method thereof. Background Technology

[0003] In general, the establishment of AI model-based systems is actively underway to enable everything from forecasting and analyzing abnormal events (intrusion, abandonment, collapse, fighting, arson, etc.) to real-time monitoring. Due to the human and economic damage caused by threats ranging from various safety accidents to natural disasters, the number of CCTVs is increasing every year to build a more robust social safety net. Consequently, a large amount of CCTV video data has become available, and operations are typically conducted by integrated control centers displaying individual CCTV footage sequentially. However, this method inevitably results in blind spots, as dedicated personnel are assigned many cameras, making it difficult to verify important incidents even when they occur. As the adoption of intelligent CCTVs increases to enable efficient management with fewer personnel, effective video anomaly detection technology has become essential.

[0004] Existing video anomaly detection technologies face limitations in applying supervised learning methodologies because it is difficult to secure large amounts of data during the process of collecting development video data for training, as the probability of abnormal events occurring is very low. Furthermore, existing supervised learning-based video anomaly detection methods have limitations because the cost of manually labeling each frame based on the occurrence of abnormal events is very high. The problem to be solved

[0006] The technical problem that the present invention aims to solve is to provide a device capable of effectively detecting abnormal behavior in video images, as well as a learning device and method thereof. Specifically, the present invention provides a video anomaly detection device and a learning device and method based on cross-learning of segment-unit features and hundreds-unit features, which can detect abnormal behavior from collected video data, including abnormal events, without labeling, and can precisely detect anomalies even when abnormal events occur over a relatively short period of time by comparing them with events before and after the occurrence of the abnormal event in the video.

[0007] The technical problems that the present invention aims to solve are not limited to those mentioned above, and other unmentioned technical problems will be clearly understood by those skilled in the art to which the present invention belongs from the description below. means of solving the problem

[0009] To achieve the above technical objective, one embodiment of the present invention provides a video anomaly detection device based on cross-learning of segment unit features and back unit features, comprising: a segment division unit that divides an input video image into segment units composed of a plurality of temporally consecutive frames; a feature extraction unit that generates a segment feature vector representing the spatiotemporal features of the segment from the divided segment; and an anomaly detection unit that inputs the segment feature vector into a segment neural network trained to make a judgment considering the temporal context to determine whether there is an anomaly, wherein the segment neural network is trained through cross-learning with a back neural network configured to learn the contextual relationships of a plurality of temporally consecutive segments.

[0010] To achieve the above technical objective, another embodiment of the present invention provides a learning device for cross-training a segment-unit anomaly detection model from a video image, comprising: a segment generation unit that divides an input video image into segment units composed of a plurality of temporally consecutive frames and generates segment feature vectors from the divided segments; a back generation unit that generates back feature vectors from a plurality of consecutive segment feature vectors; and a cross-learning unit comprising a segment neural network that determines whether an individual segment unit is anomaly and a back neural network that determines whether an anomaly is anomaly by considering the temporal context of a plurality of consecutive segments, wherein the cross-learning unit trains the segment neural network and the back neural network in a mutually complementary manner through pseudo-labels, and trains the segment neural network while modifying the pseudo-labels using the results of the temporal context determination of the back neural network.

[0011] To achieve the above technical objective, another embodiment of the present invention provides a method for training a segment neural network for detecting anomalies from a video image, comprising the steps of: dividing an input video image into segment units composed of a plurality of temporally consecutive frames and generating segment feature vectors from the divided segments; generating back feature vectors from a plurality of consecutive segment feature vectors; and cross-training a segment neural network that determines whether an individual segment unit is anomaly and a back neural network that determines whether an anomaly is anomaly by considering the temporal context of a plurality of consecutive segments, wherein the cross-training step is characterized in that the segment neural network and the back neural network perform mutually complementary learning through pseudo-labels, and the segment neural network is trained while modifying the pseudo-labels using the results of the temporal context determination of the back neural network. Effects of the invention

[0013] According to an embodiment of the present invention, the following effects can be obtained.

[0014] First, abnormal behavior can be detected without checking for abnormal events on a frame-by-frame basis in the collected video data, which can significantly reduce data preparation costs.

[0015] Second, by cross-learning local features at the segment level and global features at the hundred level, abnormal events occurring over a short period can also be sensitively detected by considering the temporal context.

[0016] Third, by using only the segment neural network among the two cross-trained neural networks, efficient and stable anomaly detection is possible in real-world applications.

[0017] Due to these effects, the present invention can be utilized to implement an intelligent CCTV system capable of effective anomaly detection while reducing labeling costs in real-world environments where abnormal events occur very rarely.

[0018] The effects of the present invention are not limited to the effects described above, and should be understood to include all effects that can be inferred from the configuration of the invention described in the detailed description of the invention or the claims. Brief explanation of the drawing

[0020] FIG. 1 is a block diagram showing the overall configuration of a video anomaly detection device based on cross-learning of segment-unit features and hundred-unit features according to one embodiment of the present invention. FIG. 2 is a block diagram showing a learning device for cross-training a segment-unit anomaly detection model from a video image according to another embodiment of the present invention. Figure 3 is a diagram for further explaining the process of learning a segment neural network by the learning device illustrated in Figure 2. Figure 4 is a diagram to further explain the segment and back neural network pseudo-label generation and cross-learning processes. FIG. 5 is a flowchart illustrating a method for cross-training a segment-unit anomaly detection model from a video image according to another embodiment of the present invention. FIG. 6 is a flowchart illustrating a specific embodiment of the learning method described in FIG. 5 by the learning device described in FIG. 2. Specific details for implementing the invention

[0021] The present invention is susceptible to various modifications and may take various forms; therefore, specific embodiments are illustrated in the drawings and described in detail in the text. However, this is not intended to limit the invention to the specific disclosed forms, and it should be understood that the invention includes all modifications, equivalents, and substitutions that fall within the spirit and scope of the invention. Similar reference numerals have been used for similar components in the description of each drawing.

[0022] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as generally understood by those skilled in the art to which the present invention pertains. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology, and should not be interpreted in an ideal or overly formal sense unless explicitly defined in this application.

[0023] Hereinafter, preferred embodiments of the present invention will be described in more detail with reference to the attached drawings.

[0024] FIG. 1 is a block diagram showing the overall configuration of a video anomaly detection device (100) based on cross-learning of segment unit features and hundred unit features according to one embodiment of the present invention.

[0025] Referring to FIG. 1, the video anomaly detection device (100) (hereinafter, video anomaly detection device) based on cross-learning of segment unit features and hundred unit features of the present embodiment includes a segment division unit (110), a feature extraction unit (120), and an anomaly detection unit (130).

[0026] The segment division unit (110) divides the input video image into segment units consisting of multiple temporally continuous frames. Here, a segment refers to a group of frames having temporal continuity, and this serves as a basic unit for capturing the temporal characteristics of abnormal behavior. For example, 16 consecutive frames can constitute one segment, and these segments are divided so as not to overlap each other.

[0027] The feature extraction unit (120) generates a segment feature vector representing the spatiotemporal characteristics of the segment from the segmented segment. The feature extraction unit (120) utilizes a feature extraction neural network pre-trained in a video behavior recognition task to generate a feature vector that effectively captures the temporal and spatial characteristics of each segment. The feature vector thus generated is used as basic data for determining abnormal behavior.

[0028] The anomaly detection unit (130) includes a learned segment neural network (131) and an anomaly judgment processing unit (132), and receives a segment feature vector as input to determine whether an anomaly exists by considering the temporal context. Here, the segment neural network (131) does not simply look at the features of individual segments, but performs the judgment by considering the contextual relationships of multiple segments that are temporally consecutive. This capability is acquired through cross-learning with a back neural network, enabling more accurate anomaly detection by considering the temporal context.

[0029] In particular, during the cross-learning process of the segment neural network (131), mutually complementary learning is performed through pseudo-labels. An initial pseudo-label is generated based on a threshold value derived from the statistical distribution of the output values ​​of the segment neural network (131), and this is modified according to the processing result of hundreds of features reflecting temporal importance through an attention mechanism. Through learning based on these modified pseudo-labels, the segment neural network (131) acquires anomaly detection capabilities that consider temporal context.

[0030] The video anomaly detection device (100) of the present invention configured in this manner can effectively detect anomalies even without labeled training data, and in particular, can accurately detect abnormal behavior occurring over a short period of time by considering the temporal context.

[0031] Below, each component is explained in more detail.

[0032] Meanwhile, to explain the feature extraction unit (120) in more detail, the feature extraction unit (120) extracts spatiotemporal features of a segment using a feature extraction neural network that has been pre-trained in a video behavior recognition task. The feature extraction neural network used at this time is pre-trained to recognize various human behaviors through a large-scale video dataset, and can generate a meaningful feature vector that includes both temporal and spatial characteristics from the input segment. Specifically, the feature extraction neural network is trained on a task of recognizing various human behaviors in a large-scale video dataset such as Kinetics or UCF101, so it can effectively capture patterns of movement over time and spatial object features. For example, it can extract features such as sudden movements or abnormal posture changes that appear in abnormal behaviors such as falling or fighting. By utilizing such a pre-trained feature extraction neural network, meaningful features of a segment can be extracted without separate labeled data for abnormal behavior detection. This effectively solves the problem of difficulty in obtaining training data because abnormal events occur very rarely in real environments.

[0033] The segment neural network (131) of the aforementioned anomaly detection unit (130) is a model trained to detect anomalies through cross-learning with the back neural network in the learning device (200). The learning of the segment neural network (131) by this learning device (200) is described in more detail in FIG. 2. The cross-learning process of the segment neural network and the back neural network in this learning device (200) is performed as follows. Statistical analysis is performed on the output values ​​of the segment neural network to calculate the mean and standard deviation of the output values, and the sum thereof is set as a threshold value. This is used as a reference point to distinguish between normal and abnormal, and an initial pseudo-label is generated by determining that a segment with an output value greater than this threshold value is abnormal and a segment with an output value smaller than this threshold value is normal. For example, if the mean of the output values ​​of the segment neural network is 0.4 and the standard deviation is 0.2, the threshold value is set to 0.6, and segments with outputs exceeding this value are determined to be abnormal. The back neural network is trained based on the generated initial pseudo-label. The back neural network performs learning by considering the temporal context of consecutive segments, and in this process, generates an attention score representing the importance of each segment. This attention score is a numerical value that quantifies how important each segment is within the overall temporal context. The attention scores generated by the back neural network are utilized to perform pseudo-label modification and segment neural network retraining. Modified pseudo-labels are generated using a weighted sum method by applying the attention scores as weights to the initial pseudo-labels. These become more sophisticated labels that reflect temporal context information, and the segment neural network is retrained using them. For example, if a specific segment receives a high attention score, its pseudo-label is reflected with a larger weight, thereby having a greater impact on the training of the segment neural network.

[0034] Based on this learning, the process of determining abnormal intervals within the segment neural network pre-configured in the detection device can be explained in more detail as follows. The composition of segments with temporal continuity processes N consecutive segments as a single unit. This serves as the basic unit for identifying the temporal context of abnormal behavior and defines the range within which the back neural network can consider the temporal context. For example, a back can be constructed by sequentially selecting N segments from a set of 16 consecutive frames. The back neural network's abnormal interval determination process comprehensively considers the output values ​​of the segment neural network for the N consecutive segments. Specifically, if even a single segment among the N segments is determined to be abnormal, the entire temporal interval containing those N segments is determined to be an abnormal interval. This enables the detection of abnormal behaviors occurring in brief moments by considering their preceding and succeeding contexts. This interval-based determination effectively reflects the characteristics of actual abnormal behaviors. For instance, abnormal behaviors such as sudden acts of violence or falling may occur over a very short period, but more accurate detection becomes possible by considering the preceding and succeeding temporal contexts. Back neural networks perform judgments that consider this temporal context, enabling them to detect abnormal behavior over a temporal continuum rather than merely determining whether individual segments are abnormal.

[0035] As described above, the anomaly judgment processing unit can process and output the anomaly judgment result of the segment neural network. The detailed operation process for this anomaly score output is as follows. The anomaly score output method of the anomaly detection unit (130) outputs an anomaly score as a real value between 0 and 1 for the segment feature vector. In this case, the closer the output value is to 1, the higher the probability that the segment contains anomaly behavior, and the closer it is to 0, the higher the probability that it is normal behavior. This probabilistic scoring system enables the reliability of anomaly behavior detection to be expressed quantitatively. Specifically, the anomaly detection unit (130) processes the feature vector of each segment through the segment neural network (131) to calculate a continuous probability value. For example, a score of 0.9 indicates a very high probability of anomaly, and a score of 0.1 indicates a very low probability of anomaly. This continuous scoring system enables the detailed expression of the degree or uncertainty of anomaly behavior. This probabilistic scoring system has great utility in actual applications. In particular, CCTV monitoring systems enable flexible operation, such as setting alarm priorities based on anomaly scores or sending notifications to administrators only when a specific threshold is exceeded. Additionally, by tracking temporal changes in anomaly scores, the occurrence and progression of abnormal behavior can be monitored.

[0036] FIG. 2 is a block diagram showing a learning device (200) that cross-learns a segment-unit anomaly detection model from a video image according to another embodiment of the present invention.

[0037] Referring to FIG. 2, the learning device (200) (hereinafter, the learning device for cross-learning the detection model) that cross-learns a segment-unit anomaly detection model from a video image of the present embodiment includes a segment generation unit (210), a back generation unit (220), and a cross-learning unit (230).

[0038] The segment generation unit (210) divides the input video image into segment units consisting of multiple temporally consecutive frames and generates segment feature vectors from the divided segments. This includes the process of generating feature vectors that capture spatiotemporal features of each segment by utilizing a feature extraction neural network pre-trained in a video behavior recognition task. Each segment serves as a basic unit for capturing the temporal characteristics of abnormal behavior.

[0039] The detailed operation process of the segment generation unit (210) is as follows.

[0040] The frame division process of the segment generation unit (210) divides the input video image into segments consisting of R frames that are temporally continuous. Here, R represents the number of frames constituting one segment, and this is set as an appropriate time unit capable of capturing the temporal characteristics of abnormal behavior. For example, if R is 16, in a video shot at 30 frames per second, each segment contains a time interval of about 0.5 seconds. This time unit is a size suitable for capturing instantaneous abnormal behavior such as falling or violence.

[0041] The feature vector generation process generates D-dimensional feature vectors by inputting the segmented segments into a feature extraction neural network pre-trained in video behavior recognition tasks. The feature extraction neural network used here is pre-trained to recognize various human behaviors on large-scale video datasets (e.g., Kinetics, UCF101, etc.), enabling the extraction of meaningful features that include both temporal and spatial characteristics. The D-dimensional feature vectors concisely represent the spatiotemporal characteristics of the segments and are subsequently utilized as foundational data for anomaly detection.

[0042] The operation of such a segment generation unit (210) enables effective feature extraction even without labeled data. By utilizing a pre-trained feature extraction neural network, meaningful features can be extracted without separate labeled training data for abnormal behavior detection, which effectively solves the problem of abnormal events occurring very rarely in real-world environments.

[0043] The back generation unit (220) generates a back feature vector from a plurality of consecutive segment feature vectors. This is intended to consider the relationship between temporally consecutive segments, and generates a back-unit feature vector by calculating the importance (attention score) of each segment through an attention mechanism and reflecting it.

[0044] The attention score calculation process of the back generation unit (220) is performed through an attention-based neural network on a series of segment feature vectors. In this process, an attention score is calculated that quantifies the temporal importance of each segment, which indicates how important a role each segment plays in the overall temporal context. For example, segments containing sudden movements or abnormal patterns receive a higher attention score.

[0045] The back feature vector generation process is performed by weighted summing multiple segment feature vectors based on calculated attention scores. Specifically, a weighted sum is conducted using the attention scores corresponding to each segment feature vector as weights, thereby generating a single back feature vector that reflects temporal context information. This generated back feature vector captures the temporal relationships between consecutive segments.

[0046] The operation of this back generation unit (220) effectively reflects the characteristics of actual abnormal behavior. For example, when abnormal behavior such as falling or violence occurs, the segments before and after that point in time have different levels of importance, which is naturally captured through the attention score. Consequently, the generated back feature vector concisely expresses this temporal context information.

[0047] The cross-learning unit (230) may include a segment neural network (231), a back neural network (232), a pseudo-label generation unit (233), a first loss function calculation unit (234), a second loss function calculation unit (235), and a learning control unit (236). Here, the learned segment neural network (231) may be used as the segment neural network (131) installed in the detection device (100) described in FIG. 1. As described above, the cross-learning unit (230) includes a segment neural network (231) that determines whether there is an anomaly at the individual segment level and a back neural network (232) that determines whether there is an anomaly by considering the temporal context of a plurality of consecutive segments. The segment neural network (231) and the back neural network (232) perform mutually complementary learning through pseudo-labels, and the segment neural network (231) is trained by modifying the pseudo-labels using the results of the temporal context determination of the back neural network (232).

[0048] Specifically, the segment neural network (231) determines whether there is an anomaly at the individual segment level, and the back neural network (232) determines whether there is an anomaly by considering the temporal context of a series of consecutive segments. These two neural networks perform mutually complementary learning through pseudo-labels, and in particular, the segment neural network (231) is trained by modifying the pseudo-labels using the results of the temporal context determination of the back neural network (232). Through this cross-learning structure, it is possible to train an effective anomaly detection model that considers temporal context even without labeled data.

[0049] The pseudo-label generation unit (233) analyzes the statistical distribution of the output values ​​of the segment neural network (231) to derive a threshold value and generates a pseudo-label based on this. Specifically, it calculates the mean and standard deviation of the output values ​​and sets the sum thereof as the threshold value, and determines whether each segment is abnormal based on this threshold value. This provides a criterion that can distinguish between normal and abnormal even without labeled data.

[0050] The detailed operation process of the pseudo-label generation unit (233) is as follows.

[0051] The segment unit determination process of the pseudo-label generation unit (233) is performed by comparing the output values ​​of the segment neural network (231) with the derived threshold value. Specifically, if the value output by the segment neural network (231) is greater than the threshold value, the segment is determined to be abnormal, and if it is less than the threshold value, it is determined to be normal, thereby generating a pseudo-label for each. At this time, the threshold value is set based on the statistical distribution of the output values, which provides a criterion for distinguishing between normal and abnormal without labeling.

[0052] The process of generating hundreds of pseudo-labels performs an integrated judgment on N consecutive segments. If even one of the N segment-unit pseudo-labels is judged to be abnormal, the entire hundreds is judged to be abnormal to generate hundreds of pseudo-labels. This reflects the characteristics of abnormal behavior occurring over temporal continuity, as abnormal behaviors such as falling or violent acts may occur in a very short moment, but it is important to consider the context before and after.

[0053] This method of generating pseudo-labels is gradually improved during the learning process. It starts with simple criteria based on statistical distributions, but as the ability of the back neural network (232) to consider temporal context improves, the reliability of the pseudo-labels also increases.

[0054] The first loss function calculation unit (234) and the second loss function calculation unit (235) each calculate a loss value for the learning of the back neural network (232) and the segment neural network (231). The first loss function calculation unit (234) calculates the difference between the output value of the back neural network (232) and the pseudo-label, and the second loss function calculation unit (235) calculates the difference between the output value of the segment neural network (231) and the pseudo-label. These two loss functions serve as indicators for determining the learning direction of each neural network.

[0055] The learning control unit (236) controls the mutually complementary learning of the two neural networks by integrally utilizing the two loss functions. In particular, it modifies pseudo-labels by utilizing the temporal context judgment results generated in the back neural network (232), and trains the segment neural network (231) based on these modified pseudo-labels. Through this process, local judgments at the segment level and global judgments at the back level complement each other and develop.

[0056] Using the modified segment-unit pseudo-labels and the output values ​​of the segment neural network (231), the segment-unit loss function calculation unit calculates the loss value through a binary cross-entropy function. Finally, each neural network is trained based on the two calculated loss function values, and this cross-training process is repeated until all segments within the video are randomly selected and used for training.

[0057] The actual anomaly detection process after the learning is completed is performed as follows. The anomaly detection unit receives a verification video and divides it so that consecutive frame bundles do not overlap, and then extracts segment-unit features through the segment generation unit (210). The segment-unit features extracted in this way are input into a segment neural network (231) that has been trained to consider temporal context through cross-learning to generate an output value between 0 and 1, and this value is used as the final anomaly detection score.

[0058] FIG. 3 is a diagram for further explaining the process of learning a segment neural network by the learning device (200) shown in FIG. 2.

[0059] First, videos containing abnormal events are collected and stored using multiple fixed cameras. Then, the stored videos are divided in a segment generation unit (210) so that consecutive frame bundles do not overlap, and each frame bundle is input into a pre-trained neural network to extract segment-unit features. The above process is performed for all stored videos to extract segment-unit features. Some m videos are selected from the entire collected video set, and N consecutive segments are selected for each of the m videos.

[0060] The detailed operation process of the bag generation unit (220) is as follows.

[0061] The attention score calculation process of the back generation unit (220) is performed through an attention-based neural network for a plurality of consecutive segment feature vectors. Specifically, as described above, the features of the selected segment unit pass through an attention-based neural network composed of two linear embedding layers in the back generation unit (220) for each of the N consecutive segments, and the importance between segments (attention score) is calculated by comparing the N consecutive segments through the following mathematical formula 1.

[0062] [Mathematical Formula 1]

[0063]

[0064] Here, W and V are learnable weight matrices for attention computation, x ij is the feature vector of the j-th segment of the i-th video, represents the normalized attention score for the k-th segment.

[0065] The attention score calculated through Equation 1 is used as a weight for each segment to generate hundreds-unit features that have the same size as a single segment-unit feature. Through this process, higher weights are assigned to segments where abnormal events occur among consecutive segments, thereby effectively extracting the features of abnormal events from the hundreds-unit features generated as a weighted sum of segment-unit features and attention scores. During this process, an attention score is calculated that quantifies the temporal importance of each segment, indicating how important a role each segment plays within the overall temporal context. For example, segments containing sudden movements or abnormal patterns receive higher attention scores.

[0066] FIG. 4 is a diagram to further explain the segment and back neural network (232) pseudo-label generation and cross-learning process.

[0067] Referring to FIG. 4, the operation process of the cross-learning unit (230) is as follows.

[0068] The cross-learning unit (230) receives and processes segment-unit features and back-unit features generated by the segment generation unit (210) and back generation unit (220), respectively. As illustrated in FIG. 4, this process generates pseudo-labels through two neural networks with the same structure and performs learning based on them.

[0069] Specifically, first, segment unit features generated from the segment generation unit (210) are input into the segment neural network (231) to predict N output values ​​for each video. This process is performed repeatedly for all m selected videos. The pseudo-label generation unit (233) calculates the mean and standard deviation of all predicted segment unit output values ​​and adds the two to set a threshold for pseudo-label generation.

[0070] Based on a set threshold, pseudo-labels for abnormal events are assigned to segments with output values ​​greater than the threshold, and pseudo-labels for normal events are assigned to segments with output values ​​smaller than the threshold. In particular, if even a single segment among the N segments contained within a video has a pseudo-label for an abnormal event, the entire video is determined to contain an abnormal event, and a pseudo-label for the abnormal event is assigned.

[0071] This segment-unit pseudo-label generation process is performed repeatedly for all m videos to generate m hundred-unit pseudo-labels. At the same time, the hundred-unit features generated by the hundred generation unit (220) are input into the hundred neural network (232) to predict m output values.

[0072] The loss function for cross-learning is calculated from two perspectives. First, in the hundreds-unit loss function calculation unit, the difference between the m output values ​​predicted by the hundreds neural network (232) and the m hundreds-unit pseudo-labels generated by the pseudo-label generation unit (233) is calculated using a binary cross-entropy function.

[0073] Second, the attention score generated in the back generation unit (220) is used to modify the segment-unit pseudo-labels generated in the previous step. This is done through Equation 2, and through this process, the pseudo-labels are modified so that a greater weight is assigned to the parts among the consecutive segments where abnormal events are likely to occur.

[0074] [Mathematical Formula 2]

[0075]

[0076] Mathematical formula 2 represents the modification process of segment-unit pseudo-labels.

[0077] Here, the left side's is the modified segment-unit pseudo-label, is the attention score for the corresponding segment, of the right side represents the original segment-unit pseudo-label, and β represents a weighting coefficient between 0 and 1. Equation 2 describes the process of generating a new pseudo-label by appropriately combining the original pseudo-label and the attention score. The reflection ratio between the original pseudo-label and the attention score can be adjusted through the value of β. For example, when β is 0.5, the original pseudo-label and the attention score are reflected with equal weight; as β approaches 0, the influence of the attention score increases, and as β approaches 1, the original pseudo-label has a greater influence.

[0078] Through this modification process, temporal context information is reflected in the pseudo-labels, and the importance of segments that are particularly likely to have abnormal events is naturally strengthened. This becomes a key mechanism that helps the segment neural network (231) learn by considering the temporal context during the cross-learning process.

[0079] The process of generating and modifying these pseudo-labels is examined in more detail for each component of the cross-learning unit (230) as follows.

[0080] The pseudo-label generation unit (233) analyzes the statistical distribution of the output values ​​of the segment neural network (231) to derive a threshold value and generates an initial pseudo-label based on this. Specifically, it calculates the mean and standard deviation of the output values ​​and sets the sum thereof as the threshold value, and determines whether each segment is abnormal based on this threshold value. This provides a criterion that can distinguish between normal and abnormal even without labeled data.

[0081] The pseudo-labels generated in this way are modified by reflecting the attention score through the process of Equation 2 described above, and the training of the two neural networks proceeds based on these modified pseudo-labels. The first loss function calculation unit (234) and the second loss function calculation unit (235) each calculate the loss value for training the back neural network (232) and the segment neural network (231).

[0082] FIG. 5 is a flowchart illustrating a method for cross-training a segment-unit anomaly detection model from a video image according to another embodiment of the present invention.

[0083] In a method for cross-training a segment-unit anomaly detection model from a video image, a video image is first received (S10), the input video image is divided into segments consisting of multiple temporally consecutive frames, and segment feature vectors are generated from the divided segments (S20). Specifically, the video image is received, and R consecutive frames are formed into a single segment in the segment generation unit (210). At this time, each segment is divided so as not to overlap to have temporal continuity. Each divided segment is input into a feature extraction neural network pre-trained in a video action recognition task and converted into a D-dimensional feature vector. This feature vector concisely represents the spatiotemporal features of the corresponding segment.

[0084] Next, a back feature vector is generated from a plurality of consecutive segment feature vectors (S30). The back generation unit (220) selects N temporally consecutive segment feature vectors from among the generated segment feature vectors. For the selected N segment feature vectors, an attention score representing the temporal importance of each segment is calculated through an attention-based neural network. Based on this attention score, the N segment feature vectors are weighted to generate a single back feature vector. Through this, the temporal contextual information of the consecutive segments is reflected in the back feature vector.

[0085] Subsequently, the segment neural network (231) and the back neural network (232) are trained in a mutually complementary manner (S40). In the cross-learning unit (230), a threshold value is first derived from the statistical distribution of the output values ​​of the segment neural network (231) to generate an initial pseudo-label. This pseudo-label is used for training the back neural network (232), and the back neural network (232) outputs a judgment result that considers the temporal context. The temporal importance information obtained from the judgment result of the back neural network (232) is used to modify the pseudo-label again, and the segment neural network (231) is trained based on the modified pseudo-label. This cross-learning process proceeds in such a way that the two neural networks utilize each other's learning results to gradually improve performance.

[0086] This cross-learning method of the present embodiment enables effective anomaly detection even without labeled data, and in particular, allows for the accurate detection of instantaneous abnormal behaviors by considering the temporal context. Furthermore, it has the advantage of enabling efficient anomaly detection using only the segment neural network after training is complete.

[0087] Explain each process in more detail.

[0088] In a method for cross-training a segment-unit anomaly detection model from video images, the processes for generating segments and generating feature vectors are specified. First, the input video image is divided into segments consisting of R frames that are temporally continuous (S21). This division process is designed to maintain temporal continuity without overlapping between segments and divides the entire flow of the video into fixed units. For example, in a video of 30 frames / second, if R is set to 16, segments of approximately 0.5 seconds are generated.

[0089] Each segment is input into a feature extraction neural network pre-trained in a video action recognition task and converted into a D-dimensional segment feature vector (S22). This feature extraction neural network is a model typically pre-trained on a large-scale video dataset (e.g., Kinetics) and can effectively extract spatiotemporal patterns within the segment. The generated D-dimensional feature vector concisely represents information such as movement, objects, and background of the corresponding segment and is subsequently utilized in the back generation unit (220) and the cross-learning unit (230).

[0090] The reason for using a pre-trained feature extraction neural network is to obtain high-quality features from the early stages of training. This is particularly important given the nature of the present invention, where the model must learn patterns on its own in unlabeled situations. Additionally, the size of the D dimension can be adjusted according to the neural network design; for example, by setting it to D=512 or D=1024, computational efficiency can be ensured while maintaining sufficient information.

[0091] The process of generating back feature vectors is described in detail as follows. The back generation unit (220) receives a plurality of consecutive segment feature vectors as input, for example, N segment feature vectors that are temporally consecutive, and processes them. First, an attention score representing the temporal importance of each segment feature vector is calculated using an attention-based neural network (S31). This attention score quantifies how important a role each segment plays in the context of the entire back, and, for example, a higher score can be assigned to a segment where abnormal behavior occurs instantaneously.

[0092] Based on the calculated attention scores, N segment feature vectors are weighted and summed to generate a single back feature vector (S32). In this process, by multiplying each segment feature vector by an attention score and summing them, the information of segments with high temporal importance is more reflected in the back feature vector. The back feature vector generated in this way concisely represents the spatiotemporal context of consecutive segments, and the back neural network (232) can utilize this to determine whether there is an anomaly considering the temporal context.

[0093] The reason for using the attention mechanism is to reflect the fact that not all segments have the same importance. For example, segments where sudden behavioral changes occur in a video are likely to be important clues for anomaly detection, so these segments are highlighted through attention scores. This method enables more sophisticated contextual analysis than simple averaging or summing methods and contributes to improving anomaly detection accuracy, especially in complex video sequences.

[0094] The aforementioned cross-learning process (S40) is explained in more detail. First, a threshold value is derived by analyzing the statistical distribution from the output values ​​of the segment neural network (231), and an initial pseudo-label is generated based on this (S41). This process is performed in the pseudo-label generation unit (233) and provides a foundation for the model to learn on its own even in the absence of labeled data. For example, the abnormal score output by the segment neural network (231) for each segment can be statistically analyzed to set a specific upper percentile as the threshold value.

[0095] The pseudo-label generation rules of the pseudo-label generation unit (233) are specifically defined. The pseudo-label generation unit (233) generates pseudo-labels based on the output values ​​of the segment neural network (231), and first applies a threshold value derived from a statistical distribution (S41-1). Segments with output values ​​greater than the threshold value are judged as abnormal, and segments with output values ​​smaller than the threshold value are judged as normal, thereby generating pseudo-labels at the individual segment level. This process provides a criterion for classifying unlabeled data during the initial learning phase.

[0096] Next, the pseudo-label is expanded to hundreds of units consisting of N consecutive segments (S41-2). Specifically, if any of the N segment-unit pseudo-labels is determined to be abnormal, the hundreds-unit pseudo-label for that hundreds is set to abnormal. Conversely, the hundreds-unit pseudo-label is generated as normal only when all segments are determined to be normal. This ensures that the abnormality of individual segments is reflected in the temporal context judgment of the hundreds, and guarantees that even momentary abnormalities are not missed.

[0097] This method is useful for increasing the sensitivity of anomaly detection. For example, even if an anomaly occurs in only one segment among consecutive segments, false negatives can be reduced by magnifying it to hundreds and recognizing it as an anomaly in the entire context. In addition, this rule provides an important criterion for the back neural network (232) to perform learning that considers temporal continuity.

[0098] Next, a first loss function is calculated to measure the difference between the output value of the back neural network (232) and the generated pseudo-label (S42). This loss function evaluates how well the back neural network (232) performs judgments considering the temporal context and is processed in the first loss function calculation unit (234). At the same time, a second loss function is calculated to measure the difference between the output value of the segment neural network (231) and the same pseudo-label (S43). This is performed in the second loss function calculation unit (235) and evaluates the anomaly judgment ability at the individual segment level.

[0099] Finally, the learning control unit (236) utilizes the first loss function and the second loss function to train the segment neural network (231) and the back neural network (232) in a mutually complementary manner (S44). Specifically, the pseudo-labels are modified by reflecting the results of the temporal context judgment of the back neural network (232), and the training of the segment neural network (231) is performed based on the modified pseudo-labels.

[0100] For example, if the back neural network (232) determines that a specific segment is incorrectly classified as abnormal, it corrects the pseudo-label of the segment to normal and reflects this in the training of the segment neural network (231) to increase the consistency between the two neural networks. This iterative cross-training gradually improves the performance of the model and, in particular, brings about harmony between the temporal context and the individual segment judgment.

[0101] The advantage of this method is that it can overcome the inaccuracy of initial pseudo-labels by dynamically modifying them during training. Additionally, since the two neural networks share the learning objective from different perspectives through the first and second loss functions, it can achieve more robust anomaly detection performance than a single neural network training method.

[0102] FIG. 6 is a flowchart illustrating a specific embodiment of the learning method described in FIG. 5 by the learning device described in FIG. 2.

[0103] Referring to FIG. 6, M videos containing abnormal events are collected and stored using multiple fixed cameras. Subsequently, the stored videos are divided in a segment generation unit (210) such that consecutive bundles of R frames do not overlap. The divided bundles of R frames are input into a neural network pre-trained in a video action recognition task using R frames as input to form segment feature vectors. Extracts feature vector It has D dimensions and corresponds to the j-th segment of the i-th video. The above process is performed for all M videos to extract a different number of segment feature vectors for each video. During training, for each batch, m videos are selected from the M videos, and N consecutive segment feature vectors are randomly extracted for each of the m videos. The extracted segment feature vectors are used in a back generation unit (220) to extract the importance between N segments using a neural network composed of weights W and V through Equation 1. The importance between segments and the segment feature vectors are weighted to generate a back-unit feature of the same size as a single segment unit feature.

[0104] Afterward, segment unit features from the segment generation unit (210) are input into the segment neural network (231) to predict N outputs for each video, and the above process is repeated for all m videos.

[0105] The pseudo-label generation unit (233) calculates the mean and standard deviation of N outputs for all m predicted videos and adds them to calculate a pseudo-label generation threshold. Segment-unit pseudo-labels for abnormal events are assigned to outputs that are large based on the threshold, and segment-unit pseudo-labels for normal events are assigned to outputs that are small. At this time, among the N pseudo-labels in the video, if there is at least one pseudo-label for an abnormal event, the pseudo-label of this video is assigned as a pseudo-label for an abnormal event; otherwise, it is assigned as a pseudo-label for a normal event.

[0106] The above process is repeated for all m videos to generate m back-unit pseudo-labels. Meanwhile, back-unit features from the back-unit generator (220) are input into the back neural network (232) to predict m outputs. Subsequently, using the predicted m outputs and the m back-unit pseudo-labels assigned by the pseudo-label generator (233), the back-unit loss function calculation unit calculates the binary cross-entropy function as the loss function. In addition, the attention score from the back-unit generator (220) and the segment-level pseudo-labels assigned in the previous step Modify the segment-unit pseudo-labels using Equation 2 to focus on the parts of consecutive segments where abnormal events are likely to occur.

[0107] Subsequently, using the modified segment-unit pseudo-labels and the output of the segment neural network (231), the segment-unit loss function calculation unit calculates the binary cross-entropy function as the loss function. Finally, each neural network is trained with the calculated loss function value. The above training process is repeated until all segments within the video are randomly selected. The anomaly detection unit collects the verification video and divides it so that consecutive bundles of R frames do not overlap, and extracts segment-unit features through the segment generation unit (210). Next, the extracted segment-unit features are input into the segment neural network (231) that has completed the training process to obtain an output value between 0 and 1, and this is considered as the anomaly detection score.

[0108] The present invention presents a method for training an anomaly detection model by dividing video images into segments and performing back-unit analysis that reflects temporal context. Attention mechanisms, dynamic pseudo-label correction, and loss function-based cross-learning enable the model to progressively improve performance even without labeled data, while guaranteeing the quality of initial training through pre-trained feature extraction. This approach is particularly suitable for instantaneous anomaly detection and efficient real-time processing.

[0109] The foregoing description of the present invention is for illustrative purposes only, and those skilled in the art will understand that other specific forms can be easily modified without altering the technical spirit or essential features of the present invention. Therefore, the embodiments described above should be understood as illustrative in all respects and not restrictive. For example, each component described as a single unit may be implemented in a distributed manner, and components described as distributed may likewise be implemented in a combined form.

[0110] The scope of the present invention is defined by the claims set forth below, and all modifications or variations derived from the meaning and scope of the claims and equivalent concepts thereof should be interpreted as being included within the scope of the present invention. Explanation of the symbols

[0112] Video anomaly detection device (100) Segment division part (110) Feature extraction unit (120) Anomaly detection unit (130) Segment neural network (131) abnormality judgment processing unit (132) Learning device (200) that cross-trains the detection model Segment creation unit (210) Bag generation unit (220) Cross-learning section (230) Segment neural network (231) Back neural network (232) Physician label generation unit (233) First loss function calculation unit (234) Second loss function calculation unit (235) Learning control unit (236)

Claims

Claim 1 A video anomaly detection device based on cross-learning of segment unit features and back unit features, comprising: a segment division unit that divides an input video image into segment units composed of a plurality of temporally continuous frames; a feature extraction unit that generates a segment feature vector representing the spatiotemporal features of the segment from the divided segment; and an anomaly detection unit that inputs the segment feature vector into a segment neural network trained to make a judgment considering the temporal context to determine whether there is an anomaly, wherein the segment neural network is characterized by being trained through cross-learning with a back neural network configured to learn the contextual relationships of a plurality of temporally continuous segments. Claim 2 A video anomaly detection device based on cross-learning of segment-unit features and back-unit features according to claim 1, wherein the cross-learning of the segment neural network and the back neural network is characterized in that the segment neural network and the back neural network perform mutually complementary learning through pseudo-labels, and the pseudo-label is generated based on a threshold value derived from the statistical distribution of output values ​​of the segment neural network for the segment feature vector, and the pseudo-label is modified using an output value obtained by inputting a back-unit feature generated by reflecting the temporal importance between the consecutive segments through an attention mechanism into the back neural network. Claim 3 A video anomaly detection device based on cross-learning of segment-unit features and back-unit features according to claim 2, wherein the cross-learning sets the threshold value as the sum of the mean and standard deviation of the output values ​​of the segment neural network, determines that a segment with an output value greater than the threshold value is abnormal and a segment with an output value smaller than the threshold value is normal, thereby generating the pseudo-label, the back neural network is trained based on the pseudo-label, and the segment neural network is trained with the modified pseudo-label obtained by modifying the pseudo-label using a weighted sum method with the attention score generated from the back neural network. Claim 4 A video anomaly detection device based on cross-learning of segment-unit features and back-unit features, wherein, in claim 2, the cross-learning is characterized in that when one or more of the output values ​​of the segment neural network for each of the plurality of segments constituting the back are determined to be anomaly, the back neural network for the corresponding back-unit feature generates an output value of anomaly. Claim 5 A video anomaly detection device based on cross-learning of segment-unit features and hundred-unit features according to claim 1, wherein the anomaly detection unit outputs an anomaly score for the segment feature vector as a value between 0 and 1, and indicates that the closer to 1, the higher the probability of an anomaly. Claim 6 A learning device for cross-training a segment neural network for detecting anomalies from video images, comprising: a segment generation unit that divides an input video image into segment units composed of a plurality of temporally consecutive frames and generates segment feature vectors from the divided segments; a back generation unit that generates back feature vectors from a plurality of consecutive segment feature vectors; and a cross-learning unit comprising a segment neural network that determines whether an anomaly exists in an individual segment unit and a back neural network that determines whether an anomaly exists by considering the temporal context of a plurality of consecutive segments, wherein the cross-learning unit is characterized in that the segment neural network and the back neural network perform mutually complementary learning through pseudo-labels, and the segment neural network is trained by modifying the pseudo-labels using the results of the temporal context determination of the back neural network. Claim 7 In claim 6, the cross-learning unit further comprises: a pseudo-label generation unit that generates a pseudo-label based on a threshold value derived from the statistical distribution of output values ​​of the segment neural network; a first loss function calculation unit that calculates a first loss function between the output value of the back neural network and the pseudo-label; a second loss function calculation unit that calculates a second loss function between the output value of the segment neural network and the pseudo-label; and a learning control unit that trains the two neural networks complementarily using the first loss function and the second loss function, modifies the pseudo-label using the temporal context judgment result of the back neural network, and trains the segment neural network based on the modified pseudo-label; characterized in that the learning device for cross-learning a segment-unit anomaly detection model from a video image. Claim 8 A learning device for cross-training a segment-unit anomaly detection model from a video image, wherein, in claim 7, the back generation unit calculates an attention score representing the temporal importance between the plurality of segment feature vectors through an attention-based neural network for the plurality of consecutive segment feature vectors, and generates the back feature vector by weighting the plurality of segment feature vectors according to the attention score. Claim 9 A learning device for cross-training a segment-unit anomaly detection model from a video image, wherein, in claim 7, the pseudo-label generation unit determines a segment having an output value greater than the threshold value as abnormal and determines a segment having an output value smaller than the threshold value as normal to generate the pseudo-label, and if one or more of the segment-unit pseudo-labels constituting the bag are determined to be abnormal, generates a bag-unit pseudo-label for the bag as abnormal. Claim 10 A method for training a segment neural network for detecting anomalies from a video image, comprising: a step of dividing an input video image into segment units composed of a plurality of temporally consecutive frames and generating segment feature vectors from the divided segments; a step of generating back feature vectors from a plurality of consecutive segment feature vectors; and a step of cross-training a segment neural network that determines whether an anomaly exists in an individual segment unit and a back neural network that determines whether an anomaly exists by considering the temporal context of a plurality of consecutive segments; wherein the cross-training step is characterized in that the segment neural network and the back neural network perform mutually complementary learning through pseudo-labels, and the segment neural network is trained while modifying the pseudo-labels using the results of the temporal context determination of the back neural network.

Images