Program, apparatus, and method for detecting anomalies in moving images.

Abstract

This invention provides a program anomaly detection device and method that detects anomalies in target video data by training with training data of video images labeled as abnormal or normal, and using captions estimated from the video images. [Solution] The anomaly detection device includes a video language model that takes video as input and generates captions for the video; an extended training database created by inputting video data of training data into the video language model and adding the generated captions to the video data of training data and labels; and an anomaly detection model trained by inputting video from the extended training database into the video language model and associating the intermediate features built internally with the labels of the video from the extended training database. In the anomaly detection stage, the anomaly detection model estimates labels corresponding to the intermediate features based on the intermediate features built internally by inputting video data of the target data into the video language model.

Description

[Technical Field]

[0001] This invention relates to a technology for detecting anomalies in video. [Background technology]

[0002] Providing people with information about dangers and risks in accordance with their surrounding circumstances is important for ensuring public safety. For example, predicting risks such as accidents and traffic jams from dashcam footage in moving vehicles, or predicting risks such as theft and crimes from surveillance camera footage, is useful.

[0003] There are three main approaches to detecting anomalies in video footage. • Unsupervised learning (UVAD: Unsupervised VAD) Weakly supervised learning (WVAD) • MLLM (Multimodal Large Language Model) based anomaly detection According to UVAD, an unsupervised learning method for detecting anomalies in video, it is necessary to collect a large amount of normal data to serve as training data. This is based on the assumption that the training data covers all modes of normality. The WVAD (Video Anomaly Detection) method, which uses weakly supervised learning for video anomaly detection, can be easily trained using weak annotations at the video level rather than at the image frame level. In contrast, MLLM-based anomaly detection has demonstrated a certain level of accuracy even with zero shots. Furthermore, considering practical monitoring applications, the accuracy of anomaly detection can be further improved by fine-tuning for specific tasks.

[0004] Conventionally, there is a technique for anomaly detection using semi-supervised learning with normal data and a small amount of abnormal data (see, for example, Patent Document 1). Specifically, an anomaly detection model is trained using one-hot representation index image data obtained by converting image data, thereby improving detection accuracy even with a small amount of abnormal data.

[0005] Furthermore, there is a technique that adds captions (explanatory text) to abnormal data and outputs detailed information when an anomaly is detected (see, for example, Patent Document 2). Specifically, a deep learning engine is trained using abnormal data with an anomaly caption added to the image of the abnormal part, and normal data with a normal caption added. [Prior art documents] [Patent Documents]

[0006] [Patent Document 1] Japanese Patent Publication No. 2023-075660 [Patent Document 2] Japanese Patent Publication No. 2024-085677 [Overview of the project] [Problems that the invention aims to solve]

[0007] In the case of UVAD, anomaly detection in video using unsupervised learning, it is not possible to guarantee that all normal data will be collected from the real world. In other words, if normal data that is not included in the training data examples is input, the system will suffer from false positives. Furthermore, in the case of WVAD, a video anomaly detection method using weakly supervised learning, and Patent Document 1, since they can only be applied to a single modal like video, they lack explanatory power for anomaly detection. Furthermore, in the case of Patent Document 2, the accuracy of the generated captions attached to the images of abnormal parts as training data greatly affects the estimation results of the deep learning engine. Specifically, the absence of information related to the anomaly affects tuning and leads to a decrease in the quality of the generated response text. Furthermore, assuming actual operation, even if a large amount of training video images can be collected, manually attaching captions to the images of their abnormal parts poses a significant constraint and requires enormous costs. Of course, captions can also be attached using a large language model, but it depends on the accuracy in zero-shot.

[0008] Therefore, an object of the present invention is to provide a program, apparatus, and method for training a multi-modal learning model of video images and language using video images with weakly assigned abnormal or normal labels as teacher data, and detecting abnormalities in the video images of target data.

Means for Solving the Problems

[0009] According to the present invention, there is provided a program for causing a computer mounted in an apparatus for detecting an abnormality in a moving image of target data to function, using teacher data with an abnormal or normal label assigned to each moving image, the program comprising: inputting a moving image and generating a caption for the moving image, a moving image language model; as a preprocessing stage, an extended teacher database in which a caption generated by inputting the moving image of the teacher data into the moving image language model is further assigned to the moving image and label of the teacher data; as a training stage, an abnormality detection model trained by associating an intermediate feature amount constructed inside by inputting the moving image of the extended teacher database into the moving image language model with the label of the moving image of the extended teacher database; and causing the computer to function accordingly; as an abnormality detection stage, based on an intermediate feature amount constructed inside by inputting the moving image of the target data into the moving image language model, causing the abnormality detection model to estimate a label corresponding to the intermediate feature amount. Characterized in that the computer is caused to function in this way.

[0010] According to another embodiment of the program of the present invention, the moving image is a video or an image, the label is a binary value representing normal or abnormal, The anomaly detection model estimates an anomaly score within a binary numerical range based on intermediate features. It is also preferable to make the computer function in this way.

[0011] According to other embodiments of the program of the present invention, The video language model is A visual encoder that converts moving images into visual feature vectors, A projector layer that projects visual feature vectors onto visual feature representations, A multimodal large-scale language model that generates captions corresponding to moving images by decoding text based on text embeddings that indicate the detection of anomalies from visual feature representations. It is also preferable to make the computer function in this way.

[0012] According to other embodiments of the program of the present invention, Anomaly detection models are This is a weakly supervised learning classifier that associates video images with labels. As part of the training phase, intermediate features built within the video language model are divided into normal and abnormal labels, and the classifier is trained using these divided intermediate features. It is also preferable to make the computer function in this way.

[0013] According to other embodiments of the program of the present invention, To take video as input and extract video clips by sorting them into normal or abnormal, a pre-trained weakly supervised learning classifier called an anomaly detection model is used. And make it work even better. As a preprocessing step, the anomaly detection model receives video footage from training data as input and outputs video clips that are selected as normal or abnormal to the video language model. It is also preferable to make the computer function in this way.

[0014] According to other embodiments of the program of the present invention, The anomaly detection model is, For each image frame in the input video, an anomaly score is estimated. The system extracts video clips containing image frames whose abnormality score satisfies predetermined conditions for abnormality, and video clips containing image frames whose abnormality score satisfies predetermined conditions for normality. It is also preferable to make the computer function in such a way.

[0015] According to other embodiments of the program of the present invention, A caption extension means that adds the anomaly score estimated by the anomaly detection model to the caption generated by the video language model to generate an extended caption, and It is also desirable to further enhance the computer's functionality.

[0016] According to other embodiments of the program of the present invention, The anomaly detection model estimates the confidence level of the anomaly score. The caption extension means adds the confidence level of the anomaly score estimated by the anomaly detection model to the caption generated by the video language model. It is also preferable to make the computer function in this way.

[0017] According to other embodiments of the program of the present invention, The caption extension means adds the type, number, and / or location of objects recognized by object recognition processing to the caption generated by the video language model for video clips containing anomalies extracted by the anomaly extraction model. It is also preferable to make the computer function in such a way.

[0018] According to other embodiments of the program of the present invention, The video language model, as part of its augmented training phase, fine-tunes itself by associating videos from an augmented training database with augmented captions. The anomaly detection model, in its augmented training phase, is trained by inputting videos from an augmented training database into a video language model and associating the internally constructed intermediate features with the labels of the videos in the augmented training database. It is also preferable to make the computer function in such a way.

[0019] According to other embodiments of the program of the present invention, The video language model, as an augmented preprocessing step, inputs video from the augmented training database, generates captions for those video, and outputs those captions to the augmented training database. The anomaly detection model, as an augmented preprocessing step, inputs video images from the augmented training database into the video language model, estimates the labels of the video images from the augmented training database using the intermediate features built internally, and outputs the labels of the video images to the augmented training database. It is also preferable to make the computer function in such a way.

[0020] According to the present invention, an anomaly detection device detects anomalies in a video of target data using training data to which each video is labeled as abnormal or normal, A video language model that takes video as input and generates captions for the video, As a preprocessing step, the video language model is input with video data from training data, and the generated captions are further added to the video data and labels in the extended training database. As a training phase, an anomaly detection model was trained by inputting videos from an augmented training database into a video language model and associating the intermediate features built internally with the labels of the videos in the augmented training database. It has, In the anomaly detection stage, the video language model is input with the video data of the target, and based on the intermediate features built up internally, the anomaly detection model estimates a label corresponding to those intermediate features. It is characterized by the following:

[0021] According to the present invention, an anomaly detection method for a device that detects anomalies in a video of target data using training data to which each video is labeled as abnormal or normal, The device, It has a video language model that takes a video as input and generates a caption for that video, As a preprocessing step, the first step involves inputting video data of training data into a video language model, generating captions, and then adding these captions to the video data and labels of the training data, and storing them in an extended training database. As a training phase, the second step involves inputting videos from an augmented training database into the video language model, associating the internally constructed intermediate features with the labels of the videos in the augmented training database, and training the anomaly detection model. In the anomaly detection stage, the video image of the target data is input into the video language model, and in the third step, the anomaly detection model trained in the second step estimates a label corresponding to the intermediate features built up internally. It is characterized by performing the following: [Effects of the Invention]

[0022] According to the program, apparatus, and method of the present invention, it is possible to train a multimodal learning model of video and language using video images with weakly labeled abnormal or normal as training data, and to detect abnormalities in the video images of target data. [Brief explanation of the drawing]

[0023] [Figure 1] This is a functional diagram of the pre-processing stage in the abnormality detection device of the present invention. [Figure 2] Figure 1 is a functional configuration diagram of the video language model in the anomaly detection device. [Figure 3] This is a functional configuration diagram of the training stage in the anomaly detection device of the present invention. [Figure 4] This is a functional configuration diagram of the abnormality detection stage in the abnormality detection device of the present invention. [Figure 5]This is a functional configuration diagram for generating extended captions in the preprocessing stage of the anomaly detection device of the present invention. [Figure 6] Figure 5 is an explanatory diagram of the video language model in the anomaly detection device. [Figure 7] Figure 5 is an explanatory diagram of the caption extension unit in the anomaly detection device. [Figure 8] This is a functional configuration diagram for retraining during the extended training phase in the anomaly detection device of the present invention. [Figure 9] This is a functional configuration diagram showing how to generate extended captions in the extended preprocessing stage of the anomaly detection device of the present invention. [Modes for carrying out the invention]

[0024] Embodiments of the present invention will be described in detail below with reference to the drawings.

[0025] Figure 1 is a functional diagram of the pre-processing stage in the anomaly detection device of the present invention.

[0026] <Pre-treatment stage> As shown in Figure 1, the anomaly detection device 1 includes a training database 100, a video language model 11, and an extended training database 101 as pre-processing steps. These functional components are implemented as programs that enable the computer installed in the device to function. Furthermore, the processing flow of these functional components can also be understood as a pre-processing step in the device's anomaly detection method.

[0027] [Teacher Database 100] The teacher database 100 stores teacher data for each video image, with each image being labeled as either abnormal or normal. Video / Image <-> Label (Normal or Abnormal) The video may be divided into clips (or instances, segments, or snippets). According to the present invention, "video" is described as a video image, but of course, it can also be implemented as a still image. The video footage may specifically be video recorded by a dashcam or video recorded by a surveillance camera. Various types of video can be applied to anomaly detection.

[0028] As shown in Figure 1, in video footage captured by a dashcam, for example, a video of a person suddenly appearing in the road is labeled with a weak level of abnormality. Here, the label is applied as a weak annotation at the video level, not at the image frame level. The label may be a binary value, for example, "Normal: 0 / Abnormal: 1".

[0029] As an example of a training database 100, for instance, it could consist of about 500 video clips, with approximately 3 / 4 being labeled as normal and 1 / 4 as being labeled as abnormal.

[0030] [Video Language Model 11] The video language model 11 takes video images as input from the training database 100 and generates captions (descriptive text) for those videos. The video-language model 11 is a Multimodal Large Language Model (MLLM), and may be, for example, LLaVA (Large Language and Vision Assistant) or GPT (Generative Pretrained Transformer). LLaVA is a model trained by Visual Instruction Tuning to generate text that explains the content of a video using a large number of video-and-text pairs.

[0031] Figure 2 is a functional diagram of the video language model in the anomaly detection device shown in Figure 1.

[0032] Figure 2 specifically illustrates the functional configuration of the LLaVA video language model 11. The video language model 11 can generate text directly from video in zero shots by integrating visual features and text embedding representations.

[0033] According to Figure 2, the LLaVA video language model 11 consists of the following three functions. • Vision Encoder Projector layer • Large-scale language models (LLMs)

[0034] The visual encoder takes video clips v from the training database 100 as input and converts them into visual feature vectors f. Video clip: v={x1, x2,···, xT} Visual feature vector: f={f1, f2,···, fT} The visual encoder processes each video clip into a high-dimensional visual feature vector ft∈R K This is done by converting the data to (K: dimension of the feature space). Specifically, a pre-trained deep learning model such as CLIP (Contrastive Language-Image Pre-Training) is used.

[0035] The projector layer projects the visual feature vector ft onto the visual feature representation Fv.

[0036] A multimodal large-scale language model generates text by decoding visual feature representations Fv based on text embedding representations Ft. The embedded text representation Ft is based on text that indicates the detection of anomalies, their precursors, or other unusual occurrences. As shown in Figure 2, it is based on text such as the following. "Does this video pose a traffic risk?" This text is a question for captioning, and to prevent the expected answer from deviating significantly, it explicitly states whether the video clip is normal or abnormal.

[0037] The multimodal large-scale language model generates natural language captions (explanatory texts) corresponding to video clips from the decoded word sequence A={y1, y2,···, yN}. P(A|Fv, Ft) = Π N n=1 P(yn|y <n, Fv, Ft)

[0038] As shown in Figure 2, the multimodal large-scale language model generates captions such as the following: "A: There are general traffic risks." **Pedestrians**: There are pedestrians on the road. It is dangerous if the driver is not paying attention. Drivers should be aware of the presence of pedestrians and maintain a sufficient distance between their vehicle and the vehicle in front.

[0039] [Extended Teacher Database 101] The extended training database 101 stores, as a preprocessing step, the video language model 11, which receives video data of training data and stores the captions generated from that data, which are then added to the video data and labels. Video / Image <-> Label (Normal or Abnormal) <-> Caption As a result, the extended training database 101 automatically adds captions to the video images, eliminating the need for manual annotation. Furthermore, if the labels in the training data are binary, such as "Normal: 0 / Abnormal: 1", the labels in the extended training database 101 will also be stored as the same binary values.

[0040] Figure 3 is a functional configuration diagram of the training stage in the anomaly detection device of the present invention.

[0041] <Training Stage> As shown in Figure 3, the anomaly detection device 1 has a video language model 11 and an anomaly detection model 12 as part of its training phase. These functional components are implemented as programs that enable the computer installed in the device to function. Furthermore, the processing flow of these functional components can also be understood as the training phase in the device's anomaly detection method.

[0042] [Video Language Model 11] The video language model 11 in the training phase is exactly the same as the video language model 11 in the preprocessing phase shown in Figure 2. The video language model 11 takes video images as input from the extended training database 101 and outputs an internal hidden layer, i.e., an intermediate feature vector Fm in the multimodal large-scale language model MLLM. This intermediate feature vector Fm is input to the anomaly detection model 12.

[0043] [Anomaly Detection Model 12] A key feature of the present invention is that the intermediate features input to the anomaly detection model 12 inherently contain textual features that represent the basis (mechanism of occurrence) of the anomaly in the video. The anomaly detection model 12 is trained by associating the intermediate feature quantities Fm output from the video language model 11 with their labels for the video images in the extended training database 101. The anomaly detection model is a weakly supervised classifier, trained as a weak (incomplete) label corresponding to the intermediate features of each video clip.

[0044] The anomaly detection model 12 may also be a classifier based on Multiple Instance Learning (MIL). Specifically, there is UR-DMU (Uncertainty Regulated Dual Memory Units). This employs a memory strategy that preserves prototypes and maximizes the separation of positive and negative examples.

[0045] As part of its training phase, the anomaly detection model 12 divides the intermediate feature vectors Fm, which are built within the video language model 11, into categories for normal or abnormal labels, and then trains the classifier using these divided intermediate feature vectors. This improves the accuracy of the classifier. Normal:B * n={F 1 m,n, F 2 m,n, ···, F m m,n} Abnormal:B* a = {F 1 m,a, F2m,a, ···, F l m,a} n: Normal moving image clip a: Abnormal moving image clip m: Number of normal moving image clips l: Number of abnormal moving image clips

[0046] Figure 4 is a functional configuration diagram of the abnormality detection stage in the abnormality detection device of the present invention.

[0047] <Abnormality detection stage> According to Figure 4, the abnormality detection device 1 detects an abnormality in the moving image of the target data as the abnormality detection stage. The moving image language model 11 inputs the moving image of the target data. Next, the moving image language model 11 outputs the intermediate feature amounts constructed inside to the abnormality detection model 12. Then, the abnormality detection model 12 estimates a label corresponding to the intermediate feature amounts.

[0048] Also, when the label in the extended teacher database 101 is binary of "normal: 0 / abnormal: 1", the abnormality detection model 12 based on semi-supervised learning can estimate a quantitatively evaluated "abnormality score". In this case, the abnormality score is a numerical value within the range of binary values from 0 to 1. The closer the abnormality score is to 1, the higher the degree of abnormality.

[0049] Figure 5 is a functional configuration diagram for generating an extended caption in the preprocessing stage in the abnormality detection device of the present invention.

[0050] The abnormality detection device 1 in Figure 5 further has an abnormality extraction model 13 and a caption extension unit 14 as compared with the preprocessing stage in Figure 1. These functional components are also realized as programs that cause a computer installed in the device to function. Also, the flow of processing of these functional components can be understood as the preprocessing stage in the abnormality detection method of the device.

[0051] <Pre-training stage> [Anomaly Detection Model 13] The anomaly detection model 13 is a weakly supervised classifier pre-trained with weak labels (normal or abnormal) assigned at the video level of the moving images. Specifically, the anomaly detection model 13 may consist of a pre-trained I3D (Two-Stream Inflated 3D ConvNet) feature extractor and the same UR-DMU as the anomaly detection model 12 described above. Here, the UR-DMU of the anomaly detection model 13 may be, for example, the same UR-DMU of the anomaly detection model 12 trained in Figure 3 described above. The anomaly detection model 12 in Figures 3 and 4 takes intermediate features as input, whereas the anomaly extraction model 13 in Figure 5 is pre-trained using video clips with pre-assigned labels. Normal:Bn={v 1 n, v 2 n, ···, v m n} Abnormal:Ba={v 1 a, v 2 a, ···, v l a}

[0052] <Pre-treatment stage> As shown in Figure 5, the anomaly detection model 13, as a preprocessing step, inputs video footage from the training database 100 and extracts key video clips that will be used for training the video language model 11. The extracted video clips are selected as either normal or abnormal.

[0053] (Extraction of key video clips) The anomaly detection model 13 extracts relatively short video clips (e.g., about 15 seconds) through the following two steps. (S1) Multiple video clips are input, and an anomaly score (normal 0 to abnormal 1) is calculated for each image frame. Because the anomaly detection model 13 is a weakly supervised learning model, the anomaly score can be estimated as a numerical range. (S2) Video clips containing image frames whose abnormality score satisfies a predetermined condition for abnormality (above the first predetermined threshold) are extracted as abnormal. Video clips containing image frames whose abnormality score satisfies a predetermined condition for normality (below the second predetermined threshold) are extracted as normal. As a result, the anomaly detection model 13 outputs the "video clips" that it has selected as normal or abnormal to the video language model 11, and also outputs their "anomaly scores" to the caption extension unit 14. By extracting key video clips from both normal and abnormal states, the recognition accuracy of the classifier in the anomaly detection model 12 can ultimately be improved during the training phase.

[0054] (Extracting key video clips) In another embodiment, the anomaly detection model 13 may estimate the anomaly score of each image frame within a relatively long period of video footage and extract the portion of the video footage corresponding to that anomaly score.

[0055] In the group of anomaly scores si,j in the video (video i, image frame j), the position j of the image frame with the "maximum" anomaly score. * This is identified as follows: j * =arg max j si,j The image frame position j * Based on this, anomaly video clips with a fixed sequence length T (e.g., 15 seconds) are extracted as key video clips vkey,i. For example, when extracting video clips that include T / 2 frames before and after a sequence length T (e.g., 7.5 seconds), this key clip is represented as follows. vkey,i=vi,j * =vi,j * -T / 2,···, vi,j * +T / 2 Furthermore, among the groups of abnormality scores si and j in the video, the position j of the image frame with the "lowest" abnormality score * From this, you can extract a normal video clip.

[0056] As another embodiment, the training of the classifier in the anomaly detection model 13 will be described as a preprocessing step. Multiple video clips are divided into normal bags and abnormal bags as follows: Normal bag: Bn={v 1 n, v 2 n, ···, v m n} Abnormal bag: Ba={v 1 a, v 2 a, ···, v l a} A ranking strategy (e.g., top-k) using anomaly scores for video clips within each bag is used to select the video clip with the highest anomaly score from the normal bag and similarly select the video clip with the lowest anomaly score from the abnormal bag, training a classifier to maximize the distance between them.

[0057] Alternatively, one could assume that the video clip with the highest anomaly score among the anomaly bags is most likely to be a true anomaly, and introduce the following Deep MIL ranking objective function. max i∈Bn s i n < 예정 i∈Ba s i a s: Anomaly score estimated by the classifier of the anomaly detection model 13 Specifically, to maximize the separation of positive and negative examples, the classifier is optimized using the following MIL ranking loss. L MIL = max(0,1 - max i∈Ba s i a + max i∈Bn s i n)

[0058] (Confidence level of anomaly scores) The anomaly detection model 13 may also estimate the confidence level (0 to 1) of the anomaly score when estimating it. The confidence level of the anomaly score is also output to the caption extension unit 14.

[0059] [Video Language Model 11] The video language model 11 generates captions for the video clips extracted by the anomaly detection model 13 and outputs these captions to the caption extension unit 14.

[0060] Figure 6 is an explanatory diagram of the video language model in the anomaly detection device shown in Figure 5.

[0061] As shown in Figure 6, the anomaly detection model 13 inputs both normal and abnormal video clips into the video language model 11. Furthermore, the following text is input as the text embedding representation Ft for the multimodal large-scale language model of the video language model 11. "Does this video pose a traffic risk?" In contrast, the following captions are extracted for both normal and abnormal video clips. (Normal) "A: There are no traffic risks." **Traffic Conditions:** Visibility is good, and road conditions are excellent. (Anomaly) "A: There are general traffic risks." **Pedestrians**: There are pedestrians on the road. It is dangerous if the driver is not paying attention. Drivers should be aware of the presence of pedestrians and maintain a sufficient distance between their vehicle and the vehicle in front.

[0062] (Estimation of captions indicating the time and severity of the anomaly) Although not shown in the diagram, you can further extend the caption by entering a text embedding expression Ft based on the following text. "Does this video pose a traffic risk?" Also, in this video, "when" and "to what extent" did the anomaly occur?

[0063] In response to this, the following caption will be added to the abnormal video clip: "A: There are general traffic risks." **Pedestrian**: "At the beginning of this video," there is a pedestrian on the road. If the driver is not paying attention, it becomes "highly" dangerous. Drivers should be aware of the presence of pedestrians and maintain a sufficient distance between their vehicle and the vehicle in front. (When: Early game, mid-game, late game) (To what extent: high, medium, low)

[0064] [Caption extension section 14] The caption extension unit 14 appends the "anomaly score" estimated by the anomaly detection model 13 to the caption generated from the video language model 11. This expands the caption for that video.

[0065] Furthermore, the caption extension unit 14 may also add the "confidence level of the anomaly score" from the anomaly detection model to the caption. Furthermore, the time the video clip was shot may be added to the caption to extend its functionality. In this way, in addition to the captions generated from zero shots by the video language model 11, anomaly scores, confidence levels, and shooting times can be added in a rule-based manner to provide information about anomalies.

[0066] Figure 7 is an explanatory diagram of the caption extension unit in the anomaly detection device shown in Figure 5.

[0067] As shown in Figure 1 above, the quality of the caption depends on the accuracy of the zero shots of the video language model 11. To this end, the caption extension unit 14 extends the caption by adding an anomaly score to it.

[0068] In another embodiment, the caption extension unit 14 may add the type, number, and / or location of objects recognized by object recognition processing to the caption generated from the video language model for video clips of anomalies extracted by the anomaly extraction model 13.

[0069] The object recognition process can specifically utilize YOLO (You Only Look Once). YOLO is a fast and highly accurate deep learning algorithm that performs object detection on the entire image at once. It divides the image into a grid and detects objects in each grid cell. Each grid cell can predict the bounding box and class probability of the object.

[0070] According to Figure 7, for abnormal video clips, a caption like the following will be added: "A: There are general traffic risks." **Pedestrians**: There are pedestrians on the road. It is dangerous if the driver is not paying attention. Drivers should be aware of the presence of pedestrians and maintain a sufficient distance between their vehicle and the vehicle in front. Abnormal score: 1.0 Human object: 1 person"

[0071] Figure 8 is a functional configuration diagram for retraining during the extended training phase of the anomaly detection device of the present invention.

[0072] <Extended training phase> As shown in Figure 8, the video language model 11 fine-tunes the model by associating videos from the extended training database 101 with their extended captions. This maximizes the likelihood of generating true word sequences (captions). The video language model 11 then outputs intermediate features based on the video to the anomaly detection model 12. The anomaly detection model 12 is trained by associating the finely tuned intermediate features with the labels of the videos in the extended training database 101. As an augmented training phase, the anomaly detection model 12 is trained by inputting videos from the augmented training database into the video language model and associating the intermediate features built internally with the labels of the videos in the augmented training database. In this way, the accuracy of anomaly detection can be improved through additional learning in which the video language model 11 and the anomaly detection model 12 are trained in a complementary manner.

[0073] Here, for fine-tuning the video language model 11, for example, LoRA (Low-Rank Adaptation) may be used. This allows the model's behavior to be adjusted without directly changing the parameters of the original model by injecting a low-rank matrix into the model's weight matrix. In particular, it can speed up the training process by updating fewer parameters than the original model.

[0074] Figure 9 is a functional configuration diagram showing how to generate extended captions in the extended preprocessing stage of the anomaly detection device of the present invention.

[0075] <Extended pre-processing stage> The video language model 11 receives video footage from the extended training database 101, generates captions for the video footage, and outputs the captions for the video footage back to the extended training database 101. Furthermore, the anomaly detection model 12 inputs video images from the extended training database 101 into the video language model 11. In response, the video language model 11 outputs internally constructed intermediate features to the anomaly detection model 12. The anomaly detection model 12 estimates the labels of the video images in the extended training database 101 and outputs the labels of the video images to the extended training database 101.

[0076] By repeating Figures 8 and 9, the video language model 11 is retrained with augmented captions, and the anomaly detection model 12 is also retrained with its intermediate features.

[0077] In addition, as another embodiment, the anomaly extraction model 13 shown in Figure 5 above is composed of an I3D feature extractor and a UR-DMU, but it can also be composed of a video language model 11 and a UR-DMU.

[0078] <Generating video images for the training database 100> In other embodiments, the video and labels in the training database 100 may be pseudo-generated data. Specifically, they may be video generated from text using Stable Diffusion or GAN (Generative Adversarial Network) as decoders. It is also preferable to include noise in the generated video. This allows for the generation of video from descriptive text and further training of the anomaly detection model 12.

[0079] As described in detail above, according to the program, apparatus, and method of the present invention, it is possible to train a multimodal learning model of video and language using video images with weakly labeled abnormal or normal as training data, and to detect abnormalities in the video images of target data.

[0080] According to the present invention, from the perspective of the video language model -> anomaly detection model, the anomaly detection model is trained to determine the presence or absence of anomalies using the internal features of the video language model. Furthermore, from the perspective of anomaly detection model -> video language model, the video language model is trained only on key video images that have been extracted as normal or abnormal by the anomaly detection model. At this time, the captions estimated from those video images are extended by the anomaly score of the anomaly detection model, and the anomaly detection model is trained from its intermediate features. In this way, the accuracy of anomaly detection can be improved by additional training that complementarily trains the anomaly extraction model, the video language model, and the anomaly detection model. This makes it possible to robustly identify even videos that are difficult to distinguish.

[0081] Furthermore, this will enable, for example, the detection of anomalies from various video images, thereby contributing to Goal 3 of the United Nations-led Sustainable Development Goals (SDGs), which is "Ensure healthy lives and promote well-being for all at all ages."

[0082] Various changes, modifications, and omissions to the scope of the technical concept and viewpoint of the present invention can be readily made by those skilled in the art with respect to the various embodiments of the present invention described above. The above description is merely illustrative and is not intended to limit the present invention in any way. The present invention is limited only to what is limited by the claims and their equivalents. [Explanation of Symbols]

[0083] 1. Anomaly detection device 100 Teacher Databases 101 Extended Teacher Database 11. Video Language Model 12 Anomaly Detection Models 13. Anomaly Detection Model 14 Caption Extension Section

Claims

1. A program that operates a computer installed in a device that detects abnormalities in video footage of target data, using training data to which each video footage is labeled as abnormal or normal, A video language model that takes video as input and generates captions for the video, As a preprocessing step, the video language model is input with video data from training data, and the generated captions are further added to the video data and labels in the extended training database. As a training phase, an anomaly detection model was trained by inputting videos from an augmented training database into a video language model and associating the intermediate features built internally with the labels of the videos in the augmented training database. and make it work In the anomaly detection stage, the video language model is input with the video data of the target, and based on the intermediate features built up internally, the anomaly detection model estimates a label corresponding to those intermediate features. A program characterized by causing a computer to function in a certain way.

2. A video or image is a video or an image. The label is a binary value representing normal or abnormal. The anomaly detection model estimates an anomaly score within a binary numerical range based on intermediate features. The program according to claim 1, characterized in that it causes the computer to function in such a way.

3. The video language model is A visual encoder that converts moving images into visual feature vectors, A projector layer that projects visual feature vectors onto visual feature representations, A multimodal large-scale language model that generates captions corresponding to moving images by decoding text based on text embeddings that indicate the detection of anomalies from visual feature representations. The program according to claim 1, characterized in that it causes a computer to function.

4. Anomaly detection models are This is a weakly supervised learning classifier that associates video images with labels. As part of the training phase, intermediate features built within the video language model are divided into normal and abnormal labels, and the classifier is trained using these divided intermediate features. The program according to claim 1, characterized in that it causes the computer to function in such a way.

5. To take video as input and extract video clips by sorting them into normal or abnormal, a pre-trained weakly supervised learning classifier called an anomaly detection model is used. And make it work even better. As a preprocessing step, the anomaly detection model receives video footage from training data as input and outputs video clips that are selected as normal or abnormal to the video language model. The program according to claim 1, characterized in that it causes the computer to function in such a way.

6. The anomaly detection model is, For each image frame in the input video, an anomaly score is estimated. The system extracts video clips containing image frames whose abnormality score satisfies predetermined conditions for abnormality, and video clips containing image frames whose abnormality score satisfies predetermined conditions for normality. The program according to claim 5, characterized in that it further causes the computer to function in such a way.

7. A caption extension means that adds the anomaly score estimated by the anomaly detection model to the caption generated by the video language model to generate an extended caption, and The program according to claim 5, characterized in that it further enables the computer to function.

8. The anomaly detection model estimates the confidence level of the anomaly score. The caption extension means adds the confidence level of the anomaly score estimated by the anomaly detection model to the caption generated by the video language model. The program according to claim 7, characterized in that it causes the computer to function in such a way.

9. The caption extension means adds the type, number, and / or location of objects recognized by object recognition processing to the caption generated by the video language model for video clips containing anomalies extracted by the anomaly extraction model. The program according to claim 7, characterized in that it further causes the computer to function in such a way.

10. The video language model, as part of its augmented training phase, fine-tunes itself by associating videos from an augmented training database with augmented captions. The anomaly detection model, in its augmented training phase, is trained by inputting videos from an augmented training database into a video language model and associating the internally constructed intermediate features with the labels of the videos in the augmented training database. The program according to claim 7, characterized in that it further causes the computer to function in such a way.

11. The video language model, as an augmented preprocessing step, inputs video from the augmented training database, generates captions for those video, and outputs those captions to the augmented training database. The anomaly detection model, as an augmented preprocessing step, inputs video images from the augmented training database into the video language model, estimates the labels of the video images from the augmented training database using the intermediate features built internally, and outputs the labels of the video images to the augmented training database. The program according to claim 10, characterized in that it further enables the computer to function in this manner.

12. An anomaly detection device that detects anomalies in video footage of target data using training data to which each video footage is labeled as abnormal or normal, A video language model that takes video as input and generates captions for the video, As a preprocessing step, the video language model is input with video data from training data, and the generated captions are further added to the video data and labels in the extended training database. As a training phase, an anomaly detection model was trained by inputting videos from an augmented training database into a video language model and associating the intermediate features built internally with the labels of the videos in the augmented training database. It has, In the anomaly detection stage, the video language model is input with the video data of the target, and based on the intermediate features built up internally, the anomaly detection model estimates a label corresponding to those intermediate features. An anomaly detection device characterized by the following features.

13. An anomaly detection method for a device that detects anomalies in video footage of target data using training data to which each video footage is labeled as abnormal or normal, The device, It has a video language model that takes a video as input and generates a caption for that video, As a preprocessing step, the first step involves inputting video data of training data into a video language model, generating captions, and then adding these captions to the video data and labels of the training data, and storing them in an extended training database. As a training phase, the second step involves inputting videos from an augmented training database into the video language model, associating the internally constructed intermediate features with the labels of the videos in the augmented training database, and training the anomaly detection model. In the anomaly detection stage, the video image of the target data is input into the video language model, and in the third step, the anomaly detection model trained in the second step estimates a label corresponding to the intermediate features built up internally. An anomaly detection method characterized by performing the following.

Images