System

Abstract

To provide a system capable of quickly detecting abnormality and grasping an abnormal event.SOLUTION: A system includes: an image caption generation model trained to generate an event captured as an image as a caption; and an abnormality detection model that makes a caption generated by inputting an image capturing a normal event to the image caption generation model as learning data and determines whether or not the caption indicates an abnormal event. The image caption generation model generates, when an image is input, a caption for an input image that is input. The abnormality detection model determines, based on the caption for the input image, whether an event captured as the input image is an abnormal event and outputs the caption for the input image in a case where it is determined that the event captured as the input image is an abnormal event.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to a system. [Background technology]

[0002] Conventionally, a label-free anomaly detection method is known in which a machine learning model is created using only normal training data without setting labels to the data, and anomalies are determined from feature differences, which are the differences from the trained normal data.

[0003] Furthermore, Patent Document 1 discloses a labeled method for detecting abnormal postures by having a machine learning model learn using skeletal data labeled by a user.

[0004] Patent Document 2 also discloses an anomaly detection device that includes an image input unit to which frame images of an observation target area are input, a detection unit that detects objects captured in the frame images input to the image input unit using a detection model constructed by machine learning using training data including the frame images of the observation target area, a determination unit that determines whether an abnormal event has occurred in the observation target area based on the object detected by the detection unit, an output unit that outputs this information if the determination unit determines that an abnormal event has occurred in the observation target area, a memory unit that stores the objects detected by the detection unit, and a setting reception unit that receives settings of classification information that indicates whether or not to determine that an abnormal event has occurred in the observation target area when an object stored in the memory unit is detected, and the setting reception unit receives, as classification information, a label that indicates whether or not to determine that an abnormal event has occurred in the observation target area when the object is detected. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Publication No. 2022-055077 [Patent Document 2] Japanese Patent Publication No. 2023-050456 Summary of the Invention [Problem to be solved by the invention]

[0006] However, there is a problem in that users cannot grasp the specific abnormalities in the data that the machine learning model has determined to be abnormal, and so they need to check the images to consider how to respond.

[0007] Furthermore, the method for detecting abnormal postures described in Patent Document 1 and the abnormality detection device described in Patent Document 2 have the problem that the user has to take the trouble of labeling.

[0008] An object of the present disclosure is to provide a system that can quickly detect anomalies and grasp abnormal events. [Means for solving the problem]

[0009] In order to achieve the above object, the system of the present disclosure comprises: an image caption generation model that has been trained to generate captions for events captured as images; and an anomaly detection model that has been trained to determine whether or not the captions indicate abnormal events, using as learning data the captions generated by inputting images of normal events into the image caption generation model; Equipped with The image caption generation model generates a caption for an input image when the image is input; The anomaly detection model determines whether the event captured in the input image is an abnormal event based on the caption for the input image, and if it determines that the event captured in the input image is an abnormal event, it outputs the caption for the input image. [Effects of the Invention]

[0010] According to the present disclosure, abnormality detection and abnormal events can be quickly identified. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 is a block diagram showing functions constituting a system according to an embodiment of the present disclosure. [Figure 2] FIG. 2 is a diagram illustrating an example of the operation of the system according to the embodiment of the present disclosure. [Figure 3] FIG. 3 is a flowchart showing an example of the operation of the system according to the embodiment of the present disclosure. [Figure 4] FIG. 4 is a diagram showing an image caption generation model in the first modification of the present embodiment. [Figure 5] FIG. 5 is a diagram showing images and the like input to the image caption generation model in the first modification of the present embodiment. [Figure 6] FIG. 6 is a flowchart showing an example of the operation of the system in the first modification of the present embodiment. [Figure 7] FIG. 7 shows the scores assigned to each word. DETAILED DESCRIPTION OF THE INVENTION

[0012] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. FIG. 1 is a block diagram showing functions constituting a system according to an embodiment of the present disclosure. System 100A according to an embodiment of the present disclosure is a system that detects unpredictable abnormal events by learning normal states and determining abnormalities based on deviations from the normal state, and also displays the details of the abnormality as text information. Specifically, system 100A is a system that provides explainability by using an image caption generation model that explains the contents of the image in an anomaly detection scenario using RGB images, and detecting an abnormality from the generated caption.

[0013] The system 100A includes a control device 100 and a storage unit 140. The control device 100 functions as an acquisition unit 110, an image caption generation model 120, and an anomaly detection model 130. In FIG. 1, each functional block shows a functional configuration rather than a hardware (device) configuration. Therefore, the functional blocks shown in FIG. 1 may be implemented in a single device, or may be implemented separately in multiple devices. Data may be exchanged between functional blocks via any means, such as a data bus.

[0014] The control device 100 receives images from a surveillance camera (not shown) via a communication line such as the Internet. The surveillance camera is installed, for example, in a factory and is capable of capturing events in the imaging area as images. The images are time-series data (video) in which multiple frames are arranged in the time direction. The acquisition unit 110 acquires the images received by the control device 100.

[0015] The storage unit 140 has a database that can read and write various information. The database is configured with a single or multiple HDDs (Hard Disk Drives), SSDs (Solid State Drives), or the like.

[0016] The memory unit 140 includes a ROM (Read Only Memory) that stores the BIOS (Basic Input Output System) of the computer that realizes the control device 100, a RAM (Random Access Memory) that serves as the working area of ​​the control device 100, an OS (Operating System), application programs, and a HDD that stores data referenced when the application programs are executed.

[0017] The storage unit 140 stores images (hereinafter referred to as input images) transmitted from the monitoring camera and acquired by the acquisition unit 110. In the following description, the input images and learning images may be collectively referred to as "images."

[0018] The storage unit 140 is allocated a predetermined amount of bytes for storing images.

[0019] The storage unit 140 stores captions, which are explanatory text for explaining events captured as images. A predetermined amount of bytes is allocated to the storage unit 140 for storing captions. The storage unit 140 stores the captions in association with the images.

[0020] The image caption generation model 120 generates captions to explain normal events for training images that capture normal events. Here, a caption is a sentence that explains, from the image, the subject, the subject's actions, the object, and one or more relationships that relate the subject, actions, and object to one another. The image caption generation model 120 also generates a sentence vector that numerically represents the generated caption. In the following description, captions for training images may be referred to as "normal captions." Note that, to improve the anomaly detection accuracy of the anomaly detection model 130, it is preferable to prepare as many training images as possible.

[0021] In the following description, training images may be referred to as "normal images." Captions for training images may be referred to as "normal captions." Normal images and normal captions may be referred to as "normal data." The image caption generation model 120 may utilize an existing open-source trained model, or may be trained using independently created training data and models.

[0022] The image caption generation model 120 generates captions for the input images in the same way that it generates captions for the training images.

[0023] The anomaly detection model 130 is trained to determine whether or not an event captured in a training image is an abnormal event, using the same training images as those used by the image caption generation model 120. Specifically, the anomaly detection model 130 is trained to determine whether or not an event captured in a training image is an abnormal event, based on the captions generated by the image caption generation model 120. In other words, the anomaly detection model 130 stores the sentence vectors extracted from the training images in the storage unit 140 as normal sentence vectors.

[0024] As with the training images, the anomaly detection model 130 also determines whether an event captured in an input image is an anomalous event based on a sentence vector extracted from the input image. In the following description, the sentence vector extracted from the input image is referred to as the "input image sentence vector." The anomaly detection model 130 compares the input image sentence vector with a normal sentence vector and determines whether the event captured in the input image is an anomalous event based on the comparison result. Specifically, if the feature difference, which is the difference between the input image sentence vector and the normal sentence vector, exceeds a threshold, the anomaly detection model 130 determines that the event captured in the input image is an anomalous event, and if the feature difference is within the threshold, the anomaly detection model 130 determines that the event captured in the input image is a normal event.

[0025] Next, with reference to FIG. 2, the operations of the image caption generation model 120 and the anomaly detection model 130 will be described, divided into a preparatory step and an anomaly detection step. FIG. 2 is a diagram illustrating an example of the operation of a system according to an embodiment of the present disclosure. Here, the "preparatory step" refers to a stage in which the trained image caption generation model 120 generates captions for training images, and the anomaly detection model 130 performs training to determine whether an event captured as an image is an abnormal event based on the caption for the input training image. Furthermore, the "anomaly detection step" refers to a stage in which the image caption generation model 120 generates a caption for an input image, and the anomaly detection model 130 determines whether the event captured as an image is a description of an abnormal event based on the caption for the input image.

[0026] <Preparation steps> An example of a caption for a training image is shown in Figure 2. For example, the image caption generation model 120 identifies the subject "man," the subject's action "carrying," the object "baggage," and the relationship "warehouse" from the training image, and generates a caption for the training image: "A man is carrying luggage in the warehouse."

[0027] Furthermore, the image caption generation model 120 identifies the subject "man", the subject's action "walking", and the relationship "warehouse" from the training image, and generates a caption for the training image: "A man is walking in the warehouse."

[0028] In this way, the image caption generation model 120 identifies subjects and the like from the input training images and generates captions for the training images. The generated captions for the training images are input to the anomaly detection model 130.

[0029] When a caption for a training image is input, the anomaly detection model 130 determines whether the event captured in the training image is an anomalous event based on the input caption. Specifically, the anomaly detection model 130 determines whether the event captured in the input caption is an anomalous event based on whether the feature difference between the input image sentence vector and the normal sentence vector exceeds a threshold.

[0030] When a caption for a training image is input, the anomaly detection model 130 determines that the event captured in the input caption is a normal event because the feature difference is within the threshold. For example, if the feature difference exceeds the threshold, the threshold is adjusted to be larger by the amount of the difference. This ensures that all feature differences between the caption and the training image are within the threshold. The anomaly detection model 130 also stores the sentence vector extracted from the training image as a normal sentence vector in the storage unit 140, associating it with the training image.

[0031] <Anomaly detection step> FIG. 2 shows an example of a caption for an input image. For example, when an image is input, the image caption generation model 120 identifies the subject "male," the subject's action "falling," and the relationship "warehouse" from the input image, and generates a caption for the input image, "A man is falling in the warehouse." The image caption generation model 120 also stores the extracted input image sentence vector in the storage unit 140 in association with the input image. The generated caption for the input image is input to the anomaly detection model 130.

[0032] When a caption for an input image is input, the anomaly detection model 130 determines whether the event captured in the input image is an abnormal event based on the input caption. Specifically, if the difference in features between the input image sentence vector and the normal sentence vector exceeds a threshold, the anomaly detection model 130 determines that the event captured in the input caption is an abnormal event. Furthermore, if the difference in features between the input image sentence vector and the normal sentence vector is within the threshold, the anomaly detection model 130 determines that the event captured in the input caption is a normal event. In the example of FIG. 2 , the input image is determined to be an abnormal event based on the difference in features between the subject actions of the training image ("carrying" and "walking") and the subject action of the input image ("falling"), and the caption stored in the storage unit 140 is output. Note that if the difference in features of the input caption is within the threshold, the anomaly detection model 130 may associate the input image sentence vector extracted from the input image with the input image as a normal sentence vector and store it in the storage unit 140, or may delete the input image and caption stored in the storage unit 140.

[0033] Next, the operation of the system according to the embodiment of the present disclosure will be described with reference to Fig. 3. Fig. 3 is a flowchart showing an example of the operation of the system according to the embodiment of the present disclosure. This flow starts when an image is input to the system 100A.

[0034] First, in step S100, the acquisition unit 110 acquires an input image. The acquired input image is stored in the storage unit 140.

[0035] Next, in step S110, the control device 100 reads out the input image from the storage unit 140, and inputs the read input image into the image caption generation model 120. The image caption generation model 120 generates a caption for the input image.

[0036] Next, in step S120, the control device 100 inputs a caption for the input image to the anomaly detection model 130.

[0037] Next, the anomaly detection model 130 determines whether the event captured in the input image is an abnormal event based on the caption for the input image (S130: Anomaly Determination). If the event captured in the input image is an abnormal event (S130: YES), the process proceeds to step S140. If the event captured in the input image is a normal event (S130: NO), the process returns to before step S100.

[0038] In step S140, the anomaly detection model 130 displays a caption for the abnormal event captured in the input image, for example, on a monitor (caption display).

[0039] System 100A in the above embodiment comprises image caption generation model 120 that has been trained to generate captions for events captured as images, and anomaly detection model 130 that has been trained to use captions generated by inputting images of normal events into the image caption generation model as learning data and to determine whether the caption indicates an abnormal event. When an image is input, image caption generation model 120 generates a caption for the input image, and anomaly detection model 130 determines whether the event captured as the input image is an abnormal event based on the caption for the input image, and if it determines that the event captured as the input image is an abnormal event, it outputs a caption for the input image.

[0040] With the above configuration, normal images that capture normal events are used as learning data to learn normal conditions, and abnormalities are determined based on the difference in features between the captions for normal images and the captions for input images, thereby detecting abnormal events that cannot be predicted, making it possible to quickly detect and grasp abnormal events.In addition, if an abnormality is determined, a caption for the input image is output, allowing the user to consider how to respond based on the output caption.

[0041] (Variation 1) Next, a modified example of this embodiment will be described. In the following description of the modified example, differences from the above embodiment will be mainly described, and the same components will be assigned the same reference numerals and descriptions thereof will be omitted.

[0042] First, a system 100A according to a first modification of the present embodiment will be described with reference to Figures 4, 5 and 6. Figure 4 is a diagram showing an image caption generation model according to the first modification of the present embodiment.

[0043] The anomaly detection model 130 also determines whether an event captured in an input image is an abnormal event based on language features extracted from the input image, similar to the training image. In the following description, the language features extracted from the input image are referred to as "input image language features." The anomaly detection model 130 compares the input image language features with normal language features and determines whether the event captured in the input image is an abnormal event based on the comparison result. Specifically, if the feature difference between the input image language features and the normal language features exceeds a threshold, the anomaly detection model 130 determines that the event captured in the input image is an abnormal event. If the feature difference is within the threshold, the anomaly detection model 130 determines that the event captured in the input image is a normal event. If the feature difference of the input caption exceeds the threshold, the anomaly detection model 130 inputs the input image language features extracted from the input image into the feature-to-text conversion model 122, which then outputs the converted text data.

[0044] An example of an input image, a language feature extraction model 121, and a feature-to-text conversion model 122 will be described with reference to Fig. 5. Fig. 5 is a diagram showing an image and the like input to an image caption generation model in Modification 1 of the present embodiment.

[0045] The linguistic feature extraction model 121 extracts linguistic features, which are the meanings of words that describe an image, from an input image. In the example shown in FIG. 5, the linguistic feature extraction model 121 extracts from the input image the subject "dog," the subject's action "living," and the associations "living room" and "indoors." The feature-to-text conversion model 122 converts the linguistic features into text. In this case, the feature-to-text conversion model 122 generates a caption such as "The dog is living in the living room."

[0046] Next, an example of the operation of the system 100A in the first modification will be described with reference to Fig. 6. Fig. 6 is a flowchart showing an example of the operation of the system in the first modification of the present embodiment. Note that the same operations as those of the system 100A in the above embodiment are given the same step numbers and their description will be omitted. This flow starts when an image is input to the system 100A.

[0047] In step S112, the language feature extraction model 121 extracts language features from the input image.

[0048] Next, in step S122, the control device 100 inputs the linguistic feature amount for the input image to the anomaly detection model 130.

[0049] If an abnormality is determined in step S130 (step S130: YES), the process proceeds to step S132. If an abnormality is not determined (step S130: NO), the process returns to before step S100.

[0050] Next, in step S132, the feature-to-text conversion model 122 converts the language features into text data.

[0051] Next, in step S140, the anomaly detection model 130 displays the text data as a caption for the input image (caption display).

[0052] In the present disclosure, the anomaly detection model 130 may also have a function to further display keywords from the caption that have a high degree of contribution to anomaly detection, thereby enabling the user to easily understand the basis for anomaly detection.

[0053] For example, in the case of the caption "A man has collapsed in a warehouse," which is given as an example, it is divided into "man," "a," "warehouse," "at," and "collapsed." A score is assigned to these five words. Figure 7 shows the score assigned to each word. Of the five words, "collapsed" is deemed to have the highest contribution to determining an abnormality, and is therefore displayed as a keyword.

[0054] Furthermore, although the above-described embodiment and modifications are directed to images, the present invention may also be directed to video, which is time-series data in which a plurality of frames capturing an event are arranged in the time direction.

[0055] Furthermore, the above-described embodiments are merely examples of specific embodiments for carrying out the present disclosure, and the technical scope of the present disclosure should not be interpreted as being limited by these embodiments. In other words, the present disclosure can be carried out in various forms without departing from its gist or main features. [Industrial Applicability]

[0056] The present disclosure is suitably used for a management device equipped with a system that is required to quickly detect anomalies and grasp abnormal events. [Explanation of symbols]

[0057] 100 control device 100A System 110 Acquisition Department 120 Image Caption Generation Model 121 Language Feature Extraction Model 122 Feature-based text conversion model 130 Anomaly Detection Model 140 Storage section

Claims

1. an image caption generation model that has been trained to generate captions for events captured as images; and an anomaly detection model that has been trained to determine whether or not the captions indicate abnormal events, using as learning data the captions generated by inputting images of normal events into the image caption generation model; Equipped with The image caption generation model generates a caption for an input image when the image is input; the anomaly detection model determines whether the event captured as the input image is an anomalous event based on the caption for the input image, and outputs the caption for the input image when it determines that the event captured as the input image is an anomalous event. system.

2. The image caption generation model is a language feature extraction model that extracts language features based on the input image; a feature-to-text conversion model for converting the extracted linguistic features into text; and outputting the converted text as a caption; The system of claim 1 .

3. The language feature extraction model extracts language features for an input image when the image is input, the anomaly detection model determines whether the event captured in the input image is an anomalous event based on the linguistic features for the input image, and if it determines that the event captured in the input image is an anomalous event, inputs the linguistic features into the feature-to-text conversion model and outputs the obtained text as a caption. The system of claim 2 .

4. When the anomaly detection model determines that the event captured in the input image is an anomalous event based on the caption for the input image, the anomaly detection model further outputs keywords that have a high contribution to the determination. A system according to any one of claims 1 to 3.

5. the input image includes time-series data in which a plurality of frames capturing the event are arranged in a time direction; A system according to any one of claims 1 to 3.

Images