System

Abstract

To provide a system that not only detects abnormalities but also enables rapid identification of abnormal events.SOLUTION: A system comprises: an anomaly detection model that is trained to determine whether an event captured as an image is abnormal; an importance extraction unit that, if an event in an input image is determined to be abnormal, extracts important part information from the input image based on the contribution of each pixel of the input image to the abnormality determination; and an image caption generation model that generates a caption for an event determined to be abnormal based on the extracted important part information.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to a system. [Background technology]

[0002] Conventionally, a method is known in which image data is input to a caption generation model and a caption for the image is output.

[0003] Also, an anomaly detection method is known in which a machine learning model is created using only normal training data, and anomalies are detected based on differences in features between the trained normal data and the model.

[0004] Furthermore, Patent Document 1 discloses an anomaly detection device that includes an image input unit to which frame images of an observation target area are input, a detection unit that detects objects captured in the frame images input to the image input unit using a detection model constructed by machine learning using training data including the frame images of the observation target area, a determination unit that determines whether an abnormal event has occurred within the observation target area based on the object detected by the detection unit, and an output unit that outputs a message to that effect if the determination unit determines that an abnormal event has occurred within the observation target area.

[0005] Furthermore, Patent Document 2 discloses an anomaly detection device that includes an acquisition unit that acquires an image of a predetermined object among parts being transported on a transport path, the image being captured by an imaging device; a division unit that divides the image portion of the object in the image acquired by the acquisition unit into multiple images; a determination unit that executes an abnormality determination process on the multiple images divided by the division unit to obtain a determination result indicating the presence or absence of an abnormality; and a display control unit that superimposes the determination result by the determination unit on the captured image and displays it on a display device.

[0006] Furthermore, Patent Document 3 discloses an anomaly detection method characterized by including: a normal learning process in which normal learning, which is machine learning using training images that serve as positive examples, is performed by a generative adversarial network consisting of a competing generative network and a discriminative network, with a skip connection model arranged in the generative network, to generate a normal model that models normal images that are training images that serve as positive examples; an evaluation process in which the degree of abnormality of the image to be determined is calculated based on an output value of the normal model obtained by inputting the image to be determined to the normal model; and a determination process in which it is determined whether the image to be determined is abnormal or not based on the degree of abnormality. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] Japanese Patent Publication No. 2023-50456 [Patent Document 2] Japanese Patent Publication No. 2023-35620 [Patent Document 3] Japanese Patent Publication No. 2022-37623 Summary of the Invention [Problem to be solved by the invention]

[0008] The anomaly detection devices and anomaly detection methods described in Patent Documents 1 to 3 are devices and methods that detect anomalies based on objects, captured images, and images to be judged, respectively. However, as the amount of data for the object described in Patent Document 1, the captured image described in Patent Document 2, and the image to be judged described in Patent Document 3 increases, it becomes difficult to obtain an accurate caption that will lead to the understanding of an abnormal event when the image to be judged is input into an image caption generation model. An object of the present disclosure is to provide a system that not only detects anomalies but also enables rapid identification of abnormal events. [Means for solving the problem]

[0009] To achieve the above objectives, the system of the present disclosure comprises: an anomaly detection model that is trained to determine whether an event captured as an image is abnormal; an importance extraction unit that, when the event in the input image is determined to be abnormal, extracts important part information from the input image based on the contribution of each pixel of the input image to the abnormality determination; an image caption generation model that generates a caption for the event determined to be abnormal based on the extracted important part information; Equipped with. [Effects of the Invention]

[0010] According to the present disclosure, in addition to detecting an abnormality, it is possible to quickly grasp an abnormal event. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 is a block diagram showing functions constituting a system according to an embodiment of the present disclosure. [Figure 2] FIG. 2 is a diagram showing an example of contribution degrees assigned to each of four pixels when the input image is divided into four pixel units. [Figure 3] FIG. 3 is a diagram showing important part information extracted from an input image. [Figure 4] FIG. 4 is a block diagram illustrating the functions of an image caption generation model according to an embodiment of the present disclosure. [Figure 5] FIG. 5 is a flowchart showing an example of the operation of the system according to the embodiment of the present disclosure. [Figure 6] FIG. 6 is a block diagram showing the functions of an image caption generation model in the first modification of the present embodiment. [Figure 7] FIG. 7 is a flowchart showing an example of the operation of the system in the first modification of the present embodiment. [Figure 8]FIG. 8 is a block diagram showing the functions of an image caption generation model in the second modification of the present embodiment. [Figure 9] FIG. 9 is a block diagram showing the functions of the text creation processing unit and the like in the second modification of the present embodiment. [Figure 10] FIG. 10 is a flowchart showing an example of the operation of the system in the second modification of the present embodiment. [Figure 11] FIG. 11 is a diagram showing weights assigned to each of a plurality of frames arranged in the time direction. [Figure 12A] FIG. 12A is a diagram showing an example of contribution degrees assigned to each unit when the input image is divided into units of four pixels. [Figure 12B] FIG. 12B is a diagram showing an example of contribution degrees assigned to each unit when the input image is divided into units of 9 pixels. DETAILED DESCRIPTION OF THE INVENTION

[0012] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. Fig. 1 is a block diagram showing functions constituting a system according to an embodiment of the present disclosure. System 100A according to an embodiment of the present disclosure is a system that determines whether an input image captures an abnormal event, and generates a caption if it is determined that an abnormal event is captured.

[0013] The system 100A includes a control device 100 and a storage unit 150. The control device 100 functions as an acquisition unit 110, an anomaly detection model 120, an importance extraction unit 130, and an image caption generation model 140. In FIG. 1, each functional block shows a functional configuration rather than a hardware (device) configuration. Therefore, the functional blocks shown in FIG. 1 may be implemented in a single device, or may be implemented separately in multiple devices. Data may be exchanged between functional blocks via any means, such as a data bus.

[0014] The control device 100 receives images from a surveillance camera (not shown) via a communication line such as the Internet. The surveillance camera is installed, for example, in a factory and is capable of capturing events in the imaging area as images. The images are time-series data (video) in which multiple frames are arranged in the time direction. The acquisition unit 110 acquires the images received by the control device 100.

[0015] The storage unit 150 has a database that can read and write various information. The database is configured with one or more HDDs (Hard Disk Drives), SSDs (Solid State Drives), or the like.

[0016] The memory unit 150 includes a ROM (Read Only Memory) that stores the BIOS (Basic Input Output System) of the computer that realizes the control device 100, a RAM (Random Access Memory) that serves as the working area of ​​the control device 100, an OS (Operating System), application programs, and a HDD that stores data referenced when the application programs are executed.

[0017] The storage unit 150 stores images (hereinafter, input images) transmitted from a surveillance camera and acquired by the acquisition unit 110. In order to reduce the amount of data stored in the storage unit 150, it may be possible to determine whether or not an input image captures an abnormal event, and store the input image if it is determined that an abnormal event is captured. Alternatively, if the input image captures a normal event, the image stored in the storage unit 150 may be deleted. In the following description, the input image and learning images (described later) may be collectively referred to as "images."

[0018] Each pixel value of an image is stored in the storage unit 150. A predetermined amount of bytes is allocated to the storage unit 150 for storing pixel values. The pixel value includes a plurality of feature amounts possessed by the pixel.

[0019] The anomaly detection model 120 prepares training images (hereinafter referred to as training images) that capture normal events, and positions the training images on a coordinate system with each of multiple feature quantities as a dimension. As a result, a normal image set is formed on the coordinate system, which is a set of multiple training images that are close to each other. Note that, in order to improve the judgment accuracy of the anomaly detection model 120, it is preferable to prepare as many training images as possible. Furthermore, in consideration of the effort required for users to assign labels, it is preferable to use an anomaly detection model without labels.

[0020] The anomaly detection model 120 positions an input image on a coordinate system in the same way as the training image. The anomaly detection model 120 determines whether an event captured in an input image is an abnormal event or a normal event based on the input image set distance, which is the distance between the input image and the normal image set on the coordinate system. Specifically, if the input image set distance is longer than a predetermined distance, the anomaly detection model 120 determines that the event captured in the image is an abnormal event. If the input image set distance is shorter than the predetermined distance or if the input image is located within the set of normal images, the anomaly detection model 120 determines that the event captured in the image is a normal event. Note that the predetermined distance may be adjustable.

[0021] Anomaly detection model 120 reads out an input image from storage unit 150 and determines whether the event captured in the read input image is an abnormal event or a normal event. If anomaly detection model 120 determines that the event captured in the input image is an abnormal event, it outputs the determination result to importance extraction unit 130.

[0022] The importance extraction unit 130 includes a calculation unit 131 and an extraction unit 132. The calculation unit 131 calculates the contribution of the feature amount of each pixel of the input image for which an event has been determined to be an abnormal event. Here, the contribution amount refers to the degree to which the feature amount of each pixel of the input image contributes to the abnormality determination.

[0023] The calculation unit 131 assigns a large contribution rate to pixels having a feature that contributes to increasing the distance between the sets of input images. Conversely, it assigns a small contribution rate to pixels having a feature that contributes to shortening the distance between the sets of input images or to positioning the input image within the set of aggregated images. The contribution rate is expressed, for example, by a probability distribution. In the following explanation, the contribution rate will be explained as a numerical value. The calculation unit 131 assigns a numerical value to each pixel of the input image according to the contribution rate.

[0024] For ease of explanation, an example of contribution degrees assigned to each of four pixels will be described below. Fig. 2 is a diagram showing an example of contribution degrees assigned to each of four pixels when the input image is divided into units of four pixels. As shown in Fig. 2, the calculation unit 131 assigns a contribution degree to each of the multiple pixels (four pixels in this case) that make up the unit. Note that the calculation unit 131 assigns a contribution degree to each of the multiple pixels, with the sum of the contribution degrees assigned to each of the multiple pixels that make up the unit being (1.0).

[0025] As shown in FIG. 2, the calculation unit 131 assigns a contribution of 0.4 to a pixel having a feature that most contributed to increasing the distance between the sets of input images among the four pixels constituting the unit. The calculation unit 131 also assigns a contribution of 0.3 to a pixel having a feature that most contributed to increasing the distance between the sets of input images among the four pixels. The calculation unit 131 also assigns a contribution of 0.2 to a pixel having a feature that most contributed to increasing the distance between the sets of input images among the four pixels. The calculation unit 131 also assigns a contribution of 0.1 to a pixel having a feature that most contributed to increasing the distance between the sets of input images among the four pixels, in other words, a feature that most contributed to shortening the distance between the sets of input images among the four pixels, or a feature that most contributed to positioning the input image within the set of normal images among the four pixels. Of the contribution degrees assigned to each of the four pixels, the contribution degree "0.4" is the largest numerical value, the contribution degree "0.3" is the second largest numerical value, the contribution degree "0.2" is the third largest numerical value, and the contribution degree "0.1" is the fourth largest numerical value. In the following description, the contribution degrees assigned as above will be represented as contribution degrees {"0.1", "0.2", "0.4", "0.3"}. Note that the contribution degrees (numerical values) listed above are merely examples. For example, the contribution degrees assigned by the calculation unit 131 to each of the four pixels constituting a unit may be contribution degrees {"0.1", "0.15", "0.5", "0.25"}.

[0026] In the present disclosure, four pixels are used as one unit, but this is not limiting. For example, nine pixels, 16 pixels, or the entire input image may be used as one unit. In this case, too, the calculation unit 131 assigns a contribution to each of the multiple pixels (9 pixels or 16 pixels) constituting the unit, with the sum of the contributions assigned to each of the multiple pixels constituting the unit being (1.0). In this case, the contributions assigned by the calculation unit 131 to each of the nine pixels constituting the unit are, for example, contributions {"0.1", "0.1", "0.1", "0.1", "0.1", "0.1", "0.1", "0.1", "0.2"}.

[0027] The extraction unit 132 extracts important part information from the input image based on the contribution of each pixel calculated by the calculation unit 131.

[0028] 3 is a diagram showing important part information extracted from an input image. As shown in FIG. 3, the extraction unit 132 extracts pixels having a relatively high contribution rate (for example, a contribution rate of 0.3 or more) from the contribution rates assigned to each pixel by the calculation unit 131 as important part information.

[0029] FIG. 4 is a block diagram illustrating the functions of an image caption generation model according to an embodiment of the present disclosure. As shown in FIGS. 1 and 4, the important part information extracted by the extraction unit 132 and the input image are input to the image caption generation model 140. The image caption generation model 140 generates a caption for the abnormal event captured as important part information based on the input image and important part information. When a description of the abnormal event is output, the user can review the description and consider how to respond. Furthermore, important part information, which is an important part within the image, is extracted, and a caption is generated using the extracted important part information, thereby enabling the generation of more accurate captions. The image caption generation model 140 may use an existing open-source trained model, or it may use a model that has been trained by creating training data and a model.

[0030] Next, an example of the operation of the system 100A according to the embodiment of the present disclosure will be described with reference to Fig. 5. Fig. 5 is a flowchart showing an example of the operation of the system according to the embodiment of the present disclosure. This flow starts when the system 100A is started.

[0031] First, in step S100, the acquisition unit 110 acquires an image received from a monitoring camera by the control device 100. The acquired image is stored in the storage unit 150.

[0032] Next, in step S110, the control device 100 inputs the acquired image to the anomaly detection model 120.

[0033] Next, in step S120, the anomaly detection model 120 determines whether the event captured in the image is an abnormal event or a normal event (anomaly determination). If the event is determined to be an abnormal event (step S120: YES), the process proceeds to step S130. If the event is determined to be a normal event (step S120: NO), the process returns to before step S100.

[0034] In step S130, the importance extraction unit 130 extracts important part information from the input image (importance part extraction).

[0035] Next, in step S140, the control device 100 inputs the important part information into the image caption generation model 140.

[0036] Next, in step S150, the image caption generation model 140 generates a caption (generates a description of the abnormal event) based on the input image and the important part information, and then this flow ends.

[0037] The system 100A in the above embodiment includes an anomaly detection model 120 that has been trained to determine whether an event captured as an image is abnormal or not, an importance extraction unit 130 that extracts important part information from the input image based on the contribution of each pixel of the input image to the abnormality determination when an event in the input image is determined to be abnormal, and an image caption generation model 140 that generates a caption for the event determined to be abnormal based on the extracted important part information.

[0038] With the above configuration, an anomaly is determined based on the contribution assigned to each pixel, allowing the basis for the anomaly to be understood with high resolution. Since the image caption generation model 140 generates a caption for an event determined to be an anomaly based on important part information, it is possible to quickly identify an abnormal event compared to simply generating a caption for an event determined to be an anomaly based on the input image. Furthermore, since captions are generated after extracting important information from the image, more accurate captions can be obtained. Furthermore, by combining the label-free anomaly detection model 120 and the image caption generation model 140, it is possible to identify an abnormal event without the user having to laboriously label the image or visually confirm the image.

[0039] (Variation 1) Next, a modified example of this embodiment will be described with reference to Figures 6 and 7. In the following description of the modified example, differences from the above embodiment will be mainly described, and the same components will be denoted by the same reference numerals and descriptions thereof will be omitted.

[0040] FIG. 6 is a block diagram showing the functions of an image caption generation model in Modification 1 of this embodiment. In Modification 1, the control device 100 further includes a pre-processing unit that performs pre-processing of the input image based on the input image and important part information. Here, "pre-processing" refers to a masking process that covers unnecessary parts of the input image that are not related to the important part information, or a trimming process that removes unnecessary parts that are not related to the important part information from the input image. Note that the pre-processing unit may perform additional weighting on the important part information as pre-processing.

[0041] The preprocessed input image and important part information are input to the image caption generation model 140. The image caption generation model 140 generates a caption based on the preprocessed input image and important part information. This enables faster anomaly detection and understanding of abnormal events. Furthermore, more accurate captions can be obtained.

[0042] Next, an example of the operation of the system 100A in Modification 1 of the present embodiment will be described with reference to Fig. 7. Fig. 7 is a flowchart showing an example of the operation of the system in Modification 1 of the present embodiment. As shown in Fig. 7, in Modification 1, when the same processing as in the above embodiment is performed, the same step numbers as those shown in Fig. 5 are assigned and their description will be omitted. In Modification 1, pre-processing (masking processing and trimming processing) shown in step S131 is performed between step S130 and step S140.

[0043] (Variation 2) Fig. 8 is a block diagram showing the functions of an image caption generation model in Modification 2 of this embodiment. Fig. 9 is a block diagram showing the functions of a sentence creation processing unit and the like in Modification 2 of this embodiment. Fig. 9 shows the numerical values ​​"0", "200", and "150" assigned to correspond to the positions of pixels in the image, and also shows the area of ​​0-150 vertically and 0-200 horizontally as an important part in the image.

[0044] In the second modification, the control device 100 further includes a text creation processing unit that performs text creation processing based on important portion information. Here, the "text creation processing" refers to processing that creates a sentence such as "Please explain the area from 0-150 pixels vertically and 0-200 pixels horizontally of the image" based on the image and important portion, as shown in Fig. 9, for example.

[0045] Next, an example of the operation of the system 100A in Modification 2 of the present embodiment will be described with reference to Fig. 10. Fig. 10 is a flowchart showing an example of the operation of the system in Modification 2 of the present embodiment. As shown in Fig. 10, when the same processing as in the above embodiment is performed in Modification 2, the same step numbers as those shown in Fig. 5 are assigned and their description will be omitted. In Modification 2, a sentence creation processing shown in step S132 is performed between step S130 and step S140.

[0046] (Variation 3) FIG. 11 is a diagram showing weights assigned to each of a plurality of frames arranged in the time direction. Although this embodiment is directed to images, in the case of a video, which is time-series data in which multiple frames capturing an event are arranged in the time direction, the above-described extraction of important part information may be performed for each frame, and important part information may also be extracted in the time direction. This makes it possible to weight information (for example, events captured in images) with high precision.

[0047] As an example, FIG. 11 shows a portion of three frames (four pixels constituting a unit) arranged in the time direction. A contribution level is assigned to each of the four pixels. FIG. 11 also shows weights of "0.1," "0.2," "0.4," and "0.3" assigned to each of the three frames. In this example, importance extraction unit 130 extracts the frame assigned a weight of "0.4."

[0048] Furthermore, in the above embodiment and modified example, anomaly detection model 120 is an unlabeled anomaly detection model in consideration of the labor required for users to assign labels, but an anomaly detection model with a label may also be used. In this case, for example, the amount of information increases by combining the label used when an anomaly is determined to exist with the sentence of modified example 2, so that an anomaly detection model 120 without a label and an anomaly detection model with a label may be used selectively depending on the situation.

[0049] Furthermore, in the above embodiment and the above modification, the importance extraction unit 130 is provided separately from the image caption generation model 140 , but it may be integrated with the image caption generation model 140 .

[0050] Furthermore, in the above embodiment and modified example, as shown in FIG. 2, the calculation unit 131 assigns a contribution rate to each of a plurality of pixels constituting a unit. In the present disclosure, the method of assigning contribution rates is not limited to the method shown in FIG. 2. The calculation unit 131 may assign a contribution rate for each unit. FIG. 12A is a diagram showing an example of the contribution rate assigned for each unit when the input image is divided into units of four pixels. FIG. 12B is a diagram showing an example of the contribution rate assigned for each unit when the input image is divided into units of nine pixels. By assigning a contribution rate to each unit composed of a plurality of pixels, the amount of calculation can be reduced, thereby enabling faster anomaly detection and understanding of abnormal events.

[0051] Furthermore, in the above embodiment and modified example, the input image is positioned on a coordinate system whose dimensions are the feature amounts possessed by each pixel of the input image, and the contribution of the feature amount is defined as the contribution amount that contributes to increasing the inter-input image set distance, which is the distance between the input image and the set of normal images; in other words, the contribution amount that contributes to an abnormality determination in which an event captured by the input image is determined to be an abnormal event; however, the feature amounts and contribution amounts in the present disclosure are not limited to this.

[0052] The following describes an example in which the abnormal event is a fire. If the feature contains information indicating a color belonging to the red family and information indicating a brightness equal to or greater than a threshold, the anomaly detection model 120 determines that the event captured in the image is a fire (abnormal event) based on these feature values, for example, by referring to a decision tree algorithm.

[0053] As with the training images, the anomaly detection model 120 also determines whether an event in an input image is an abnormal event or a normal event. That is, the anomaly detection model 120 reads the input image from the storage unit 150 and determines whether the event captured in the read input image is an abnormal event or a normal event. If the anomaly detection model 120 determines that the event captured in the input image is an abnormal event, it outputs the determination result to the importance extraction unit 130.

[0054] The calculation unit 131 of the importance extraction unit 130 calculates the contribution of the feature amount of each pixel of the input image for which an event has been determined to be an abnormal event. The calculation unit 131 assigns a large contribution to pixels that exhibit a color close to a color belonging to the reddish group. Conversely, the calculation unit 131 assigns a small contribution to pixels that exhibit a color far from a color belonging to the reddish group. Furthermore, the calculation unit 131 assigns a large contribution to pixels that exhibit a high brightness that exceeds a threshold. Conversely, the calculation unit 131 assigns a small contribution to pixels that exhibit a low brightness that is below the threshold.

[0055] For example, the calculation unit 131 assigns a large contribution of "0.4" to a pixel that indicates a color close to a reddish color and that indicates a high brightness that exceeds a threshold. The calculation unit 131 also assigns the next largest contribution of "0.3" to a pixel that indicates a color close to a reddish color and that indicates a low brightness that is less than the threshold. The calculation unit 131 also assigns the next largest contribution of "0.2" to a pixel that indicates a color that is far from a reddish color and that indicates a high brightness that exceeds the threshold. The calculation unit 131 also assigns the next largest contribution of "0.1" to a pixel that indicates a color that is far from a reddish color and that indicates a low brightness that is less than the threshold. In this case, the calculation unit 131 assigns a contribution to each of the multiple pixels, with the sum of the contributions assigned to the multiple pixels that make up the unit being (1.0).

[0056] The extraction unit 132 extracts important part information from the input image based on the contribution of each pixel. The image caption generation model 140 generates a caption for the abnormal event based on the important part information. For example, the image caption generation model 140 generates a caption such as "There may be a fire at XX."

[0057] In this way, the user may define the conditions for determining an abnormality, and the degree of agreement with the conditions may be treated as the degree of contribution.

[0058] Furthermore, the above-described embodiments are merely examples of specific embodiments for carrying out the present disclosure, and the technical scope of the present disclosure should not be interpreted as being limited by these embodiments. In other words, the present disclosure can be carried out in various forms without departing from its gist or main features. [Industrial Applicability]

[0059] The present disclosure is suitably used for a management device equipped with a system that is required to not only detect anomalies but also quickly grasp abnormal events. [Explanation of symbols]

[0060] 100 control device 100A System 110 Acquisition Department 120 Anomaly Detection Model 130 Importance extraction part 131 Calculation Unit 132 Extraction part 140 Image Caption Generation Model 150 Storage section

Claims

1. an anomaly detection model that is trained to determine whether an event captured as an image is abnormal; an importance extraction unit that, when the event in the input image is determined to be abnormal, extracts important part information from the input image based on the contribution of each pixel of the input image to the abnormality determination; an image caption generation model that generates a caption for the event determined to be abnormal based on the extracted important part information; Equipped with system.

2. a preprocessing unit that preprocesses the input image based on the important part information; the image caption generation model generates the caption based on the preprocessed input image; The system of claim 1 .

3. a text creation processing unit that identifies the input image as a target for text creation based on the important part information; the image caption generation model generates the caption based on the identified input image; The system of claim 1 .

4. the input image has time-series data in which a plurality of frames capturing the event are arranged in a time direction; the importance extraction unit further extracts important part information and weights the important part information in the time direction for each of the plurality of frames; The system of claim 1 .

Images