Anomaly detection device, method, and program

Abstract

To provide an anomaly detection device that is robust to changes in the shooting environment. [Solution] The anomaly detection device according to the embodiment includes an acquisition unit, a generation unit, a detection unit, and an output unit. The acquisition unit acquires a first sample that is the target of anomaly detection. The generation unit uses a trained model to generate a first text representing the content of the first sample from the first sample. The detection unit determines whether or not there is an anomaly in the first sample based on statistics associated with all or part of the first text in a text information dictionary. The text information dictionary associates all or part of a second text representing the content of a second sample included in the training dataset with statistics related to the frequency of occurrence of all or part of the second text in the training dataset. The output unit outputs the result of the determination of whether or not there is an anomaly in the first sample.

Description

Technical Field

[0001] Embodiments of the present invention relate to an abnormality detection device, method, and program.

Background Art

[0002] In recent years, the need for abnormality detection using surveillance camera images and the like has been increasing. In particular, unsupervised abnormality detection has the advantages that only normal images are used for learning, abnormal images and annotations are not required during learning, and unknown abnormalities can be detected. On the other hand, when there are fluctuations in the shooting environment such as changes in the position and angle of the camera used for shooting and changes in sunlight conditions due to changes in the shooting time zone, it is difficult to perform highly accurate abnormality detection.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Non-Patent Documents

[0004]

Non-Patent Document 1

Non-Patent Document 2

[0005] The problem that this invention aims to solve is to provide an anomaly detection device, method, and program that enable highly accurate anomaly detection even under conditions where the shooting environment fluctuates. [Means for solving the problem]

[0006] An anomaly detection device according to an embodiment includes an acquisition unit, a generation unit, a detection unit, and an output unit. The acquisition unit acquires a first sample to be anomaly detected. The generation unit uses a trained model to generate a first text representing the content of the first sample from the first sample. The detection unit determines whether or not the first sample is anomaly based on statistics associated with all or part of the first text in a text information dictionary. The text information dictionary associates all or part of a second text representing the content of a second sample included in a training dataset with statistics related to the frequency of occurrence of all or part of the second text in the training dataset. The output unit outputs the result of determining whether or not the first sample is anomaly. [Brief explanation of the drawing]

[0007] [Figure 1] A functional block diagram showing an example of an anomaly detection device according to the first embodiment. [Figure 2] A diagram showing the processing flow during learning by the anomaly detection device according to the first embodiment. [Figure 3] A figure showing an example of a training dataset according to the first embodiment. [Figure 4] A diagram showing the processing flow when an anomaly is detected by the anomaly detection device according to the first embodiment. [Figure 5] A figure showing an example of an inference dataset according to the first embodiment. [Figure 6] A diagram showing an example of a display screen for the determination result according to the first embodiment. [Figure 7] A functional block diagram showing an example of an anomaly detection device according to the third embodiment. [Figure 8] A diagram showing the processing flow during learning by the anomaly detection device according to the third embodiment. [Figure 9] A figure showing an example of a training dataset according to the third embodiment. [Figure 10] A diagram showing an example of a text information dictionary according to the third embodiment. [Figure 11] This diagram shows the processing flow when an anomaly is detected by the anomaly detection device according to the third embodiment. [Figure 12] Figure showing an example of an inference dataset according to the third embodiment. [Figure 13] Functional block diagram showing an example of an anomaly detection device according to the fourth embodiment. [Figure 14] Functional block diagram showing an example of an anomaly detection device according to the fifth embodiment. [Figure 15] Functional block diagram showing an example of an anomaly detection device according to the sixth embodiment. [Figure 16] Figure showing the flow of processing during anomaly detection by the anomaly detection device according to the sixth embodiment. [Figure 17] Figure showing an example of an inference dataset according to the sixth embodiment. [Figure 18] Figure showing an example of a zero-shot object detection result according to the sixth embodiment. [Figure 19] Figure showing an example of a display screen of an anomaly location visualization image and a determination result according to the sixth embodiment. [Figure 20] Functional block diagram showing an example of an anomaly detection device according to the seventh embodiment. [Figure 21] Figure showing the flow of processing during anomaly detection by the anomaly detection device according to the eighth embodiment. [Figure 22] Figure showing an example of a learning dataset according to the eighth embodiment. [Figure 23] Figure showing an example of an inference dataset according to the eighth embodiment. [Figure 24] Functional block diagram showing an example of an anomaly detection device according to the ninth embodiment. [Figure 25] Figure showing the flow of processing during learning by the anomaly detection device according to the ninth embodiment. [Figure 26] Figure showing an example of a learning dataset according to the ninth embodiment. [Figure 27] Figure showing the flow of processing during anomaly detection by the anomaly detection device according to the ninth embodiment. [Figure 28] Figure showing an example of an inference dataset according to the ninth embodiment. [Figure 29] Functional block diagram showing an example of an anomaly detection device according to the tenth embodiment. [Figure 30] This figure shows an example of the hardware configuration of the anomaly detection device according to this embodiment. [Modes for carrying out the invention]

[0008] The anomaly detection device, method, and program according to this embodiment will be described below with reference to the drawings.

[0009] <First Embodiment> Figure 1 is a functional block diagram showing an example of an anomaly detection device 10 according to the first embodiment. The anomaly detection device 10 is a computer that trains a text information dictionary using training samples and uses the text information dictionary to determine whether or not an anomaly exists in a sample to be detected. In this embodiment, a sample means data that can be represented as a vector or multidimensional tensor, such as image data, time-series video, or audio signals. In the following description, a sample will be assumed to be image data. Image data will also be simply referred to as an image. As shown in Figure 1, the anomaly detection device 10 has an acquisition unit 11, a text generation unit 12, a preprocessing unit 13, a statistics calculation unit 14, a storage unit 15, a text information dictionary 16, an anomaly detection unit 17, and an output unit 18.

[0010] The acquisition unit 11 acquires samples. During training, the acquisition unit 11 acquires a training dataset. The training dataset contains multiple training samples. Training samples refer to samples used to train the anomaly detection model. In the following description, it is assumed that the training dataset contains only normal samples, but the training dataset may contain both normal and abnormal samples. A normal sample refers to a sample determined to be normal by any means, and an abnormal sample refers to a sample determined to be abnormal by any means. During anomaly detection, the acquisition unit 11 acquires the sample to be detected as an anomaly. Hereinafter, the sample to be detected as an anomaly will be referred to as the target sample. Specifically, the acquisition unit 11 acquires an inference dataset containing one or more target samples. At this time, the acquisition unit 11 may acquire multiple samples together as one batch. Hereinafter, when training samples and target samples are not distinguished, they will simply be referred to as samples.

[0011] When using an outdoor surveillance camera to capture a landscape at a particular location, if the goal is to detect unusual objects as anomalies, the training dataset is assumed to contain a large number of images that do not include such objects. Similarly, if the goal is to detect defects in a subject within a given image, the training dataset is assumed to contain a large number of images of the same object that do not contain such defects.

[0012] The text generation unit 12 uses a trained model to generate text representing the content of a sample from a sample acquired by the acquisition unit 11. In this embodiment, text means character information representing the content of a sample. Specifically, text includes sentences, words, combinations of words, clauses consisting of multiple words, and dependency relationships. As the trained model, a machine learning model (hereinafter referred to as the text generation model) that has been trained to take a sample as input and output text representing the content of the sample is used. If the sample is an image, a so-called image-to-text model that converts the image into text representing the content of the image can be used as the text generation model. Specifically, as an image-to-text model, caption generation models, visual question answering (VQA) models, and multimodal large language models (LLMs) using prompts can be used. A caption generation model is, as an example, a machine learning model that has been trained to take an image as input and output a caption, which is a descriptive text of the content of the image. A visual question answering model is a machine learning model that has been trained to take a sample and a prompt, which is a question about the content of the sample, as input and output an answer to the question. Hereafter, texts based on training samples will be referred to as training texts, and texts based on target samples will be referred to as target texts. When there is no distinction between training texts and target texts, they will simply be referred to as texts.

[0013] The preprocessing unit 13 performs arbitrary preprocessing on the text generated by the text generation unit 12. Preprocessing may include, as an example, a splitting process that divides the text into multiple parts and / or an exclusion process that removes information unnecessary for anomaly detection from the text. Here, "part" means any sentence component shorter than the input text, such as a word, a combination of words, a clause or dependency relationship consisting of multiple words. Specifically, the splitting process divides the text into multiple words. Other preprocessing may include a process that determines the part of speech of each word generated by the splitting process and extracts words of a specific part of speech. Here, the specific part of speech can be set to any part of speech such as a noun, adjective, or verb. Furthermore, the specific part of speech is not limited to one type, but may be set to multiple types, for example, nouns and adjectives. Other examples of preprocessing may include a process that corrects plural nouns to singular nouns. Other examples of preprocessing may include a process that pairs adjectives with the nouns they describe. In this case, in addition to adjective-noun pairs, individual adjectives and individual nouns may also be output. For example, from the sentence "cute dog," the three sentence components "cute," "dog," and "cute dog" may be output. Another example of preprocessing is to include a process that removes duplicates and outputs the words if the same word appears more than once in a sentence.

[0014] The preprocessing unit 13 is not mandatory, and subsequent processing may be performed on the text generated by the text generation unit 12. For example, if the text generated by the text generation unit 12 consists of words rather than sentences, the preprocessing unit 13 can be omitted.

[0015] The statistics calculation unit 14 calculates statistics for all or part of the training text for each of the multiple training samples. The statistics according to this embodiment refer to indicators related to the frequency of occurrence of all or part of the training text in the training dataset. "Part of the text" means: This refers to sentence components such as words, combinations of words, clauses consisting of multiple words, and dependency relationships. For example, the statistics calculation unit 14 calculates the occurrence probability of words output by the preprocessing unit 13. Occurrence probability is an example of a statistic. Specifically, first, the statistics calculation unit 14 calculates the occurrence frequency of all or part of the learning text corresponding to each of the multiple learning samples included in the learning dataset, and calculates a statistic based on the calculated occurrence frequency. Occurrence frequency means the number of occurrences. For example, if part of the learning text is words, the words that appear in the learning text corresponding to each learning sample included in the learning dataset are counted word by word. The word count is an example of occurrence frequency. After the count for the entire learning dataset is completed, the statistics calculation unit 14 calculates the occurrence probability of each word by dividing the occurrence frequency of each word by the number of learning samples.

[0016] The statistics calculation unit 14 may count each word individually rather than each word that appears, for each combination of multiple types of words (hereinafter referred to as "word pair") contained in a single text. For example, if a single text contains three types of words, "dog," "cat," and "human," the unit may count three types of word pairs: (dog, cat), (cat, human), and (dog, human). In this case, the statistics calculation unit 14 calculates the co-occurrence probability of each word pair by dividing the frequency of occurrence of the word pair by the number of training samples. A word pair is an example of "part of the text," and co-occurrence probability is an example of a statistic.

[0017] The statistics calculation unit 14 may calculate the conditional joint probability by dividing the co-occurrence probability of a word pair by the occurrence probability of the words that make up the word pair. For example, the conditional joint probability of "dog" and "cat" can be calculated using the following equation (1). Conditional joint probability is an example of a statistic.

[0018]

number

[0019] The statistical calculation unit 14 may count the text corresponding to each training sample included in the training dataset. After the counting is complete for the entire training dataset, the statistical calculation unit 14 calculates the probability of each text appearing by dividing the frequency of each text by the number of training samples.

[0020] The memory unit 15 is a memory device that stores the text information dictionary 16. The text information dictionary 16 links all or part of the learning text, which represents the content of the learning samples included in the learning dataset, with statistics related to the frequency of occurrence of all or part of the learning text in the learning dataset. All or part of the learning text means the entire text, clauses, dependency relationships, words and / or word pairs of the learning text, and the statistics mean the probability of occurrence, co-occurrence probability and / or conditional joint probability calculated by the statistics calculation unit 14. The text information dictionary 16 is created as a table or database that links all or part of the learning text with the statistics.

[0021] The anomaly detection unit 17 determines whether a target sample is abnormal based on statistical quantities associated with all or part of the target text corresponding to the target sample in the text information dictionary 16. Specifically, the anomaly detection unit 17 calculates the degree of abnormality of the target text based on statistical quantities associated with all or part of the target text. The anomaly detection unit 17 then determines that the target sample is abnormal if the degree of abnormality is greater than a threshold, and determines that the target sample is normal if the degree of abnormality is less than the threshold. The threshold may be arbitrarily set according to user instructions, or it may be determined based on the trend of statistical quantities registered in the text information dictionary 16. Alternatively, if abnormal samples already exist, it may be determined based on the degree of abnormality of the abnormal samples.

[0022] An example of how the degree of abnormality is calculated is as follows: First, the abnormality detection unit 17 identifies a statistical quantity associated with each word or combination of words belonging to a specific part of speech contained in the target text in the text information dictionary 16. Next, the abnormality detection unit 17 calculates the word abnormality degree based on the identified statistical quantity. Then, the abnormality detection unit 17 determines that the target sample is abnormal if the maximum value of the calculated word abnormality degree is greater than the threshold, and determines that the target sample is normal if the maximum value is less than the threshold.

[0023] As an example, the anomaly detection unit 17 calculates the degree of anomaly for each word using the word sequence contained in the target text and the text information dictionary 16. For each word in the obtained word sequence, the anomaly detection unit 17 checks whether that word is included in the text information dictionary 16. If it is not included, the anomaly detection unit 17 sets the degree of anomaly for that word to 1. If the word is included in the text information dictionary 16, the anomaly detection unit 17 obtains the occurrence probability p of that word and sets the degree of anomaly for that word to 1-p. After calculating the degree of anomaly for all words in the word sequence, the anomaly detection unit 17 sets the maximum value among them as the degree of anomaly for the target sample.

[0024] If co-occurrence probabilities are stored in the text information dictionary 16, the anomaly detection unit 17 similarly calculates the degree of anomaly for word pairs within the target text. If conditional joint probabilities are stored in the text information dictionary 16, the anomaly detection unit 17 obtains the conditional joint probability p of the word pair stored in the text information dictionary 16 for the word pair contained in the target text of the target sample, and sets the degree of anomaly for the corresponding word pair to 1-p.

[0025] The output unit 18 outputs the result of the abnormality detection unit 17's determination of whether or not an abnormality exists in the target sample. The output destination of the determination result may be a display device provided on the abnormality detection device 10 or a display device of a computer connected to the abnormality detection device 10 via a network. Alternatively, the output destination of the determination result may be a storage device provided on the abnormality detection device 10 or a storage device of a computer connected to the abnormality detection device 10 via a network.

[0026] Figure 2 is a diagram showing the processing flow during learning by the anomaly detection device 10 according to the first embodiment. In the following description, the sample is assumed to be an image. Figure 3 is a diagram illustrating three normal images I11, I12, and I13, which are an example of a learning dataset according to the first embodiment.

[0027] First, the statistics calculation unit 14 initializes the word counter (step S11). A word counter is prepared for each word and is an object used to count the frequency of word occurrences. In step S11, all word counters are initialized to 0. The word counter may be prepared in advance, or it may be generated in step S15 when a new word is detected.

[0028] When step S11 is performed, the acquisition unit 11 acquires normal images from the training dataset (step S12). In step S12, normal images are acquired one at a time. When step S12 is performed, the text generation unit 12 generates text representing the content of the normal images acquired in step S12 (step S13). Specifically, the text generation unit 12 inputs the normal images into the image caption model and generates a caption representing the content of the input normal images. The caption is an example of text.

[0029] As an example, let's assume that the normal image I11 shown in Figure 3 was acquired in the initial step S12. The normal image I11 shows flowers and grass. Therefore, the caption generated for the normal image I11 would be, for example, "There are a flower and grass."

[0030] Once step S13 is performed, the preprocessing unit 13 performs preprocessing on the text generated in step S13 (step S14). Specifically, the preprocessing unit 13 performs word segmentation on the caption and extracts words belonging to nouns from the caption. In the case of the normal image I11 in Figure 3, word segmentation divides the caption into seven words: (There, are, a, flower, and, grass, .). Further preprocessing for extracting words belonging to nouns is performed, and two words, (flower, grass), are extracted from the aforementioned seven words.

[0031] When step S14 is performed, the statistics calculation unit 14 counts the words extracted in step S14 (step S15). In the case of the normal image I11 in Figure 3, the statistics calculation unit 14 adds 1 to the word counter value for the two words (flower, grass). Since the word counter value is initialized to 0, the word counter value for each word, "flower" and "grass," becomes 1.

[0032] When step S15 is performed, the acquisition unit 11 determines whether or not there are any unprocessed normal images (step S16). If it is determined that there are any unprocessed normal images (step S16: YES), steps S12 to S16 are repeated for the unprocessed normal images.

[0033] In this embodiment, since two normal images I12 and I13 remain as shown in Figure 3, the processing from steps S12 to S15 is repeated for these normal images I12 and I13 as well. For example, since only grass is shown in normal image I12, the caption for normal image I12 is "There is grass." Since both flowers and grass are shown in normal image I13, the caption for normal image I13 is "There are grass and a flower." At this point, the word counter for the word "flower" is set to 2, and the word counter for the word "grass" is set to 3.

[0034] If it is determined that there are no unprocessed normal images (Step S16: NO), the statistics calculation unit 14 calculates the word occurrence probability by dividing the word counter value by the number of normal images included in the training dataset (Step S17). In the example in Figure 3, the number of normal images is 3, so the value of each word counter is divided by 3. As a result, the occurrence probability of the word "flower" is 2 / 3, and the occurrence probability of the word "grass" is 1.

[0035] When step S17 is performed, the memory unit 18 registers the occurrence probability of the word obtained in step S14, calculated in step S17, into the text information dictionary 16 (step S18). The text information dictionary 16 associates words with the occurrence probability corresponding to those words. In the example in Figure 3, the data (flower, 2 / 3) and (grass, 1) are registered in the text information dictionary 16.

[0036] With the above steps completed, the learning process by the anomaly detection device 10 according to the first embodiment is finished.

[0037] Figure 4 is a diagram showing the processing flow when an anomaly is detected by the anomaly detection device 10 according to the first embodiment. Figure 5 is a diagram illustrating two target images I21 and I22, which are an example of an inference dataset according to the first embodiment.

[0038] First, the acquisition unit 11 acquires target images for anomaly detection from the inference dataset (step S21). In step S21, all target images included in the inference dataset may be acquired at once, or only some of the target images may be acquired.

[0039] When step S21 is performed, the text generation unit 12 generates text representing the content of the target image based on the target image acquired in step S21 (step S22). Specifically, the text generation unit 12 inputs the target image into an image caption model and generates a caption representing the content of the input target image.

[0040] For example, in Figure 5, image I21 shows grass and a bucket. Therefore, the caption generated for image I21 would be, for example, "There are grass and a bucket." In Figure 5, image I22 shows a flower. Therefore, the caption generated for image I22 would be, for example, "There is a flower."

[0041] When step S22 is performed, the preprocessing unit 13 performs preprocessing on the caption generated in step S22 (step S23). Specifically, the preprocessing unit 13 performs word segmentation on the caption and extracts words belonging to nouns from the caption. In the case of the target image I21 in Figure 5, the caption is divided into seven words: (There, are, grass, and, a, bucket, .). Further preprocessing for extracting words belonging to nouns is performed, and two words, (grass, bucket), are extracted from the aforementioned seven words. In the case of the normal image I22, its caption is divided into five words: (There, is, a, flower, .). Further preprocessing for extracting words belonging to nouns is performed, and one word, (flower), is extracted from the aforementioned five words.

[0042] When step S23 is performed, the anomaly detection unit 17 obtains the occurrence probability p from the text information dictionary 16 for each word extracted in step S23 (step S24). In the case of the normal image I21 in Figure 5, the occurrence probability of the word "grass" extracted in step S23 is a value of 1. Here, since the word "bucket" is not registered in the text information dictionary 16, its occurrence probability is set to a value of 0. In the case of the normal image I22 in Figure 5, the occurrence probability of the word "flower" extracted in step S23 is a value of 2 / 3.

[0043] When step S24 is performed, the anomaly detection unit 17 calculates the word anomaly score 1-p based on the occurrence probability p obtained in step S24 (step S25). The word anomaly score represents the degree to which the representation of the word is anomaly in the target image, in light of the overall trend of multiple normal images included in the training dataset. The word anomaly score is calculated for each of the one or more words related to a single target image. Specifically, the anomaly detection unit 17 calculates the word anomaly score 1-p by subtracting the occurrence probability p from 1. In the case of target image I21 in Figure 5, the word anomaly score for the word "grass" is 1-1=0, and the word anomaly score for the word "bucket" is 1-0=1. In the case of target image I22 in Figure 5, the word anomaly score for the word "flower" is 1-2 / 3=1 / 3.

[0044] When step S25 is performed, the anomaly detection unit 17 sets the maximum value among the one or more word anomaly scores calculated in step S25 as the image anomaly score (step S26). The image anomaly score represents the degree to which what is depicted in the target image is anomaly in relation to the overall trend of multiple normal images included in the training dataset. Only one image anomaly score is calculated for each target image. In the case of target image I21 in Figure 5, the word anomaly score for the word "bucket" is the highest at a value of 1, so the image anomaly score for target image I21 is set to a value of 1. In the case of target image I22 in Figure 5, only the word "flower" exists, so the image anomaly score is set to 1 / 3 of the word anomaly score for the word "flower".

[0045] When step S26 is performed, the anomaly detection unit 17 performs an abnormality determination of the target image based on the image abnormality degree set in step S26 (step S27). Specifically, the anomaly detection unit 27 compares the image abnormality degree to a threshold, and if the image abnormality degree is greater than the threshold, it determines that the target image is abnormal, and if the image abnormality degree is smaller than the threshold, it determines that the target image is normal. In the case of target image I21 in Figure 5, when the threshold is set to 0.5, the image abnormality degree is 1, which is greater than 0.5, so the target image is determined to be abnormal. In the case of target image I22 in Figure 5, when the threshold is set to 0.5, the image abnormality degree is 1 / 3, which is less than 0.5, so the target image is determined to be normal.

[0046] When step S27 is performed, the output unit 18 outputs the judgment result output in step S27 to the display device (step S28). As an example, the output unit 18 displays the target sample, the target text, and the judgment result side by side on the display device. In this case, the output unit 18 may display a specific word in the target text that has an image abnormality score (maximum word abnormality score) with different visual effects depending on whether the image abnormality score is greater than or less than the threshold. The output unit 18 may also display the image abnormality score alongside the specific word.

[0047] Figure 6 shows an example of a display screen I3 showing the results of determining whether or not an abnormality is present in a target image. The display screen I3 shown in Figure 6 shows, as an example, the results of determining the abnormality of the two target images I21 and I22 shown in Figure 5. As shown in Figure 6, the display screen I3 includes the target image I21, the target image I22, and the determination result display field I31. The determination result I31 displays the determination result display field I32 for target image I21 and the determination result display field I33 for target image I22. The determination result display fields I32 and I33 display, for each target image I21 and I22, the target text generated in step S22, the occurrence probability obtained in step S24, the word abnormality score calculated in step S25, the image abnormality score set in step S26, and the determination result output in step S27.

[0048] For the target text, words for which occurrence probability and word abnormality score are to be acquired should be highlighted, for example, by underlining. Specifically, for target image I21, the words "grass" and "bucket" should be highlighted by underlining, and for target image I22, the word "flower" should be highlighted by underlining. Conversely, words such as "There" and "are" are not to be acquired and therefore should not be highlighted by underlining. The occurrence probability, word abnormality score, and image abnormality score should be displayed aligned below the corresponding word. For example, for the word "grass," the occurrence probability "1" and word abnormality score "0" are displayed, and the image abnormality score is not displayed because it has not been acquired. For the word "bucket," the occurrence probability "1," word abnormality score "0," and image abnormality score "1" are displayed. For words such as "There" and "are," none of the indicators are displayed because they are not to be acquired for occurrence probability, word abnormality score, or image abnormality score.

[0049] The judgment result will display either "Normal" or "Abnormal". A threshold value may also be displayed alongside the judgment result. The words targeted for image abnormality assessment will be displayed with different visual effects depending on whether the judgment result is abnormal or normal. Specifically, since target image I21 was judged to be "Abnormal", the word "bucket", which is targeted for image abnormality assessment, will be displayed in bold. Since target image I22 was judged to be "Normal", the word "flower", which is targeted for image abnormality assessment, will be displayed in normal boldness.

[0050] As described above, by displaying the target image and the judgment result side by side, users can understand the relationship between each target image and the judgment result. By displaying the text, occurrence probability, word abnormality level, and image abnormality as the basis for the judgment result, users can understand which part of the text was used to determine the abnormality, and can also assess the accuracy of the judgment result.

[0051] Note that the display screen for the judgment results shown in Figure 6 is just an example, and the display content can be designed arbitrarily. For example, it is not necessary to display all of the text, occurrence probability, word abnormality level, and image abnormality levels, and the display of these can be arbitrarily set according to the user. In addition, the visual effect used to highlight words for which occurrence probability and word abnormality levels are obtained is not limited to underlining, but can be set to any visual effect such as display color or annotation. Furthermore, the visual effect for words for which image abnormality levels are obtained, which should differ between "abnormal" and "normal" cases, can also be set to any visual effect such as display color or annotation, not just font weight.

[0052] With the above steps completed, the processing by the abnormality detection device 10 according to the first embodiment when an abnormality is detected is finished.

[0053] As described above, the anomaly detection device 10 according to the first embodiment includes an acquisition unit 11, a text generation unit 12, an anomaly detection unit 17, and an output unit 18. The acquisition unit 11 acquires target samples that are subject to anomaly detection. The text generation unit 12 uses a trained model to generate target text representing the content of the target samples from the target samples. The anomaly detection unit 17 determines whether or not there is an anomaly in the target sample based on statistics linked to all or part of the target text in the text information dictionary 16. The text information dictionary 16 links all or part of the training text representing the content of the training samples included in the training dataset with statistics related to the frequency of occurrence of all or part of the training text in the training dataset. The output unit 18 outputs the result of the determination of whether or not there is an anomaly in the target sample.

[0054] Typical unsupervised anomaly detection methods convert normal samples, such as normal images, into features and store them as a feature dictionary. Then, they convert the sample to be detected as an anomaly into a feature, and if that feature deviates from the trend of the feature set stored in the feature dictionary, they determine that the sample is anomaly. In this way, typical unsupervised anomaly detection converts samples into features, so if the sample acquisition environment differs significantly, the differences in the acquisition environment will also be reflected in the features. Therefore, typical unsupervised anomaly detection is vulnerable to changes in the acquisition environment.

[0055] On the other hand, the text generation unit 12 according to this embodiment converts the target sample into target text using a text generation model, making it possible to convert the content of the target sample into target text, which is more abstract and excludes information unnecessary for anomaly detection. For example, if the sample is an image, even if there are fluctuations in the shooting environment such as the camera's field of view or lighting, it becomes possible to convert it into text that abstractly represents the main content of the image, which is less affected by the shooting environment. Therefore, according to this embodiment, it is possible to convert the content of the target sample into target text in a text information format that is less affected by changes in the acquisition environment. The same applies to training samples. The text information dictionary 16 links training text that represents the content of the training sample with statistics on the frequency of occurrence of the training text. That is, even if the acquisition environment of the training sample changes significantly, the text information dictionary 16 can store the content of these training samples in a text format that is robust to changes in the acquisition environment. The anomaly detection unit 17 applies the text information dictionary 16 to the target text, making it possible to determine that a target sample corresponding to text that deviates from the trend of training text stored in the text information dictionary 16 is an anomaly. This enables robust anomaly detection that is resistant to changes in the sample acquisition environment.

[0056] <Second Embodiment> The text generation unit 12 according to the second embodiment uses a prompt-based text generation model as the trained model, rather than an image caption generation model. The prompt-based text generation model takes a sample and a prompt as input and outputs text corresponding to the combination of the sample and the prompt. A prompt is text that represents an instruction to the text generation model. Prompts can be question sentences about the content of the sample, command sentences to the text generation model used to obtain text (the output of the text generation model), or other text. Hereinafter, the text generation model according to the second embodiment will be assumed to be a visual question-answering model that uses question sentences as prompts. The anomaly detection device according to the second embodiment will now be described. In the description of this embodiment, parts that are the same as those in the first embodiment described above will be omitted or simplified.

[0057] The acquisition unit 11 acquires samples in the same manner as in the first embodiment, as well as prompts to be input to the visual question-answering model. Prompts are acquired for both training samples and target samples. A prompt means a question about the content of the sample. For example, if the sample is an image, the prompt may be a sentence asking about an object in the image, such as "What is the object in the image?", a sentence specifying a point of interest or asking about the state of an object, such as "What is the state of the object in the center of the image?", or a sentence asking about the number of objects in the image, such as "How many parts are in the image?". The acquisition unit 11 may acquire multiple prompts. The same prompt may be used for all samples, but if metadata is attached to the sample, it may be changed according to the metadata.

[0058] The text generation unit 12 inputs the samples and prompts acquired by the acquisition unit 11 into the visual question answering model and generates response sentences to the prompts as text. For example, if an image of grass and the prompt "What is the object in the image?" are input into the visual question answering model, the response sentences "There is grass" or "Grass" will be output as text. If there are multiple prompts, the text generation unit 12 generates multiple response sentences for each of the multiple prompts for a single sample. In this case, the text generation unit 12 may output all of the multiple response sentences, or it may select the response sentences to be output from among the multiple response sentences based on indicators such as sentence length and the number of nouns, and output only the selected response sentences. Response sentences are acquired for both the training sample and the target sample.

[0059] The processing of the preprocessing unit 13, statistical calculation unit 14, storage unit 15, anomaly detection unit 17, and output unit 18 is the same as in the first embodiment, so a description will be omitted.

[0060] As described above, the anomaly detection device 10 according to the second embodiment can manipulate the content of the text generated by the text generation unit 12 by using prompts, making it possible to perform anomaly detection from a perspective that the user is more interested in compared to the first embodiment. For example, when using the prompt "What is the object shown in the image?", it becomes possible to perform anomaly detection from the perspective of the object shown in the image.

[0061] <Third Embodiment> The anomaly detection device according to the third embodiment converts text representing the contents of a sample into features and determines whether or not there is an anomaly in the sample based on the features. The anomaly detection device according to the third embodiment will be described below. In the description of this embodiment, parts that are the same as those of the first embodiment described above will be omitted or simplified.

[0062] Figure 7 is a functional block diagram showing an example of an anomaly detection device 20 according to the third embodiment. As shown in Figure 7, the anomaly detection device 20 includes an acquisition unit 21, a text generation unit 22, a preprocessing unit 23, a statistical calculation unit 24, a feature extraction unit 25, a storage unit 26, a text information dictionary 27, an anomaly detection unit 28, and an output unit 29. The acquisition unit 21, text generation unit 22, preprocessing unit 23, and output unit 29 are substantially the same as the acquisition unit 11, text generation unit 12, preprocessing unit 13, and output unit 18 according to the first embodiment, respectively.

[0063] The statistics calculation unit 24 calculates the frequency of occurrence of all or part of the learning text corresponding to each of the multiple learning samples included in the learning dataset, and calculates statistics based on the calculated frequency of occurrence. If "all or part of the learning text" is a word or combination of words belonging to a specific part of speech, the statistics calculation unit 24 calculates the probability of occurrence as a statistic based on the frequency of occurrence of that word or combination of words.

[0064] The feature extraction unit 25 extracts features from all or part of the text representing the content of the sample. Features are expressed in the form of scalars, vectors, or tensors. In the following explanation, it is assumed that features are vectors. Vectorized features are called feature vectors. It is assumed that all or part of the text consists of words. Existing methods such as word2vec or ELMo described in Non-Patent Literature 1 can be used as means of extracting features from all or part of the text. Alternatively, the feature extraction unit 25 may use the trained model used in the text generation unit 22 to extract features from all or part of the text.

[0065] The memory unit 26 associates all or part of the text with the statistics calculated by the statistics calculation unit 24 and the features extracted by the feature extraction unit 25, registers them in the text information dictionary 27, and stores the text information dictionary 27. For example, the text information dictionary 27 uses words as keys and stores pairs of the occurrence probability of a word and its feature vector as values.

[0066] The anomaly detection unit 28 calculates the word abnormality score for all or part of the target text based on the statistical quantities and feature quantities associated with all or part of the target text in the text information dictionary 27. The anomaly detection unit 28 determines that the target sample is abnormal if the maximum value of the calculated word abnormality scores (image abnormality score) is greater than a threshold, and determines that the normal sample is normal if the image abnormality score is less than the threshold.

[0067] If "all or part of the learning text" consists of words or combinations of words belonging to a specific part of speech, the anomaly detection unit 28 calculates the word anomaly score for that word or combination based on the occurrence probability and feature quantities associated with the words or combinations of words belonging to a specific part of speech contained in the target text in the text information dictionary 27. If "all or part of the learning text" consists of words belonging to a specific part of speech, the text information dictionary 27 registers the word as the key and the pair of the word's occurrence probability and its vectorized feature quantity as the value. The anomaly detection unit 28 calculates the word anomaly score for nouns contained in the caption of the target image using the following equation (2). Here, p(j) is the occurrence probability of word j contained in the text information dictionary 27, and d(x i , x j ) is the Euclidean distance between the feature vector of word i and the feature vector of word j.

[0068]

number

[0069] Figure 8 is a diagram showing the processing flow during learning by the anomaly detection device 20 according to the third embodiment. In the following description, the sample is assumed to be an image. Figure 9 is a diagram illustrating three normal images I41, I42, and I43, which are an example of a learning dataset according to the third embodiment.

[0070] Steps S31-37 shown in Figure 8 are the same as steps S11-17 shown in Figure 2. Since normal image I41 shown in Figure 9 shows a car, a person, and a road, the learning text based on normal image I41 will include the words "car", "person", and "road". Since normal image I42 shows a car, a person, and a road, the learning text based on normal image I42 will include the words "car", "person", and "road". Since normal image I43 shows a car and a road, the learning text based on normal image I43 will include the words "car" and "road". Then, the probability of the word "car" appearing is calculated to be 1, the probability of the word "person" appearing is calculated to be 2 / 3, and the probability of the word "road" appearing is calculated to be 1.

[0071] When step S37 is performed, the feature extraction unit 25 extracts features from each of the words that appear in the training dataset (step S38). As described above, feature vectors are calculated as features. Specific methods for calculating feature vectors based on words include obtaining high-dimensional vectors using machine learning models that convert words into vectors, such as word2vec, as mentioned earlier. In the case of normal images I41, I42, and I43 in Figure 9, feature vectors are extracted for the words "car," "person," and "road." Here, it is assumed that two-dimensional vectors (0.7, 2.7) are calculated for the word "car," (-0.1, 1.7) for the word "person," and (0.5, 2.2) for the word "road."

[0072] When step S38 is performed, the memory unit 26 registers the word obtained in step S34, the occurrence probability calculated in step S37, and the feature quantities extracted in step S38 into the text information dictionary 27 (step S39).

[0073] Figure 10 shows an example of a text information dictionary 27 according to the third embodiment. As shown in Figure 10, the text information dictionary 27 associates a word with a corresponding feature quantity x and a corresponding occurrence probability p. For example, the word "car", the feature quantities (0.7, 2.7), and the occurrence probability "1" are associated and registered in the text information dictionary 27. This makes it possible to systematically store the feature quantities along with their occurrence probabilities in the text information dictionary 27 in a searchable manner, in association with the word.

[0074] With the above steps completed, the learning process by the anomaly detection device 20 according to the third embodiment is finished.

[0075] Figure 11 is a diagram showing the processing flow when an anomaly is detected by the anomaly detection device 20 according to the third embodiment. Figure 12 is a diagram illustrating two target images I51 and I52, which are an example of an inference dataset according to the third embodiment.

[0076] Steps S41 to S43 correspond to steps S21 to S23 in Figure 4, so their explanation is omitted here. Regarding the target image I51 in Figure 12, since it shows a road and a cone, in step S42 the text "There are a corn and a road." is generated, and in step S43 the two words (corn, road) are extracted.

[0077] When step S43 is performed, the feature extraction unit 25 extracts features from each word extracted in step S43 (step S44). Specifically, the feature extraction unit 25 converts each word into a feature vector using a machine learning model that converts words into vectors, such as word2vec. The details of vectorization are the same as during training, so they are omitted here. However, since the feature vector for the word "road" was calculated during training, that result is used as is. As a result, let's assume that the feature vector for the word "corn" is (2.6, 0.8).

[0078] When step S44 is performed, the anomaly detection unit 28 calculates the word anomaly score for each word from the occurrence probability and feature vector registered in the text information dictionary 27 (step S45). Specifically, the anomaly detection unit 28 searches the text information dictionary 27 using the word as a key to read out the occurrence probability and feature vector, and calculates the word anomaly score based on the read occurrence probability and feature vector. The word anomaly score is calculated based on equation (2) above. As an example, the formula for calculating the word anomaly score (corn) for the word "corn" is given by equation (3) below. As a result, the word anomaly score (corn) is -0.19. Similarly, the word anomaly score is calculated for the word "road". The word anomaly score (road) is -1.89. A smaller word anomaly score means that the word is normal.

[0079]

number

[0080] Steps S46 to S48 are the same as steps S26 to S28 in Figure 4, so their explanation is omitted. The target image I51 contains a cone, and there are no normal images containing a cone in the training dataset. For example, by setting the threshold for word abnormality to -1.0, the image abnormality of target image I51, -0.19, is greater than the threshold, and as expected, target image I51 is determined to be abnormal.

[0081] Next, we will explain the processing of the target image I52 in Figure 12. For the target image I52, since it shows a road and a van, in step S42 the text "There are a van and a road." is generated, and in step S43 the two words (van, road) are extracted. In step S44, the word "van" is converted into a feature vector (0.8, 3.0). Here, since both vans and cars belong to the category of vehicles, it is expected that the feature vector of the word "van" will be similar to the feature vector of the word "car". In fact, as shown in Figure 10, the feature vector of the word "car" is (0.7, 2.7), which is similar to the feature vector of the word "van" (0.8, 3.0).

[0082] In step S45, the word anomaly score for the word "van" is calculated, similar to the example for the word "corn". The formula for calculating the word anomaly score (van) for the word "van" is shown in equation (4) below. The word anomaly score (van) is -1.29.

[0083]

number

[0084] The word "van" does not appear in the captions of the training dataset and is not registered in the text information dictionary 27. However, because the word "car," which has a similar feature vector, appeared during training, the word abnormality score for "van" is relatively small. Therefore, for example, by setting the threshold for word abnormality score to -1.0, the target image I52 is judged as normal, as expected, even though there are no normal images of vans in the training dataset.

[0085] According to the third embodiment, it becomes possible to detect anomalies by taking into account the relationships between conceptually similar words, such as the word "van" and the word "car."

[0086] <Fourth Embodiment> The anomaly detection device according to the fourth embodiment clusters samples and creates and references different text information dictionaries within each cluster. The anomaly detection device according to the fourth embodiment will be described below. In this description, parts that are the same as those in the first embodiment described above will be omitted or simplified.

[0087] Figure 13 is a functional block diagram showing an example of an anomaly detection device 30 according to the fourth embodiment. As shown in Figure 13, the anomaly detection device 30 includes an acquisition unit 31, a text generation unit 32, a clustering unit 33, a statistical calculation unit 34, a storage unit 35, a text information dictionary 36, a cluster determination unit 37, an anomaly detection unit 38, and an output unit 39. The acquisition unit 31, the text generation unit 32, and the output unit 38 are substantially the same as the acquisition unit 11, the text generation unit 12, and the output unit 18 according to the first embodiment, respectively.

[0088] The clustering unit 33 performs clustering on the training dataset, dividing multiple training samples into multiple clusters. The clustering unit 33 performs clustering using an unsupervised clustering method. The number of clusters may be determined manually by the user, or it may be determined automatically using some metric. The clustering unit 33 can do this by extracting features from the training dataset using a convolutional neural network or the like, and then using these features to divide multiple training samples into multiple clusters using a clustering method such as K-Means. Each training sample is assigned an identifier for the cluster to which it belongs (hereinafter referred to as cluster ID). Clustering makes it possible, for example, to place training samples acquired in similar environments into the same cluster, and training samples acquired in different environments into different clusters.

[0089] The statistics calculation unit 34 calculates statistics for each of the multiple clusters, representing the content of the learning text that belongs to that cluster.

[0090] The memory unit 35 associates the cluster ID assigned to each sample by the clustering unit 33 with all or part of the learning text and the statistics calculated by the statistics calculation unit 34. In other words, if the number of clusters is k, k text information dictionaries 36 are created.

[0091] When an anomaly is detected, the cluster discrimination unit 37 identifies the cluster to which the target sample belongs from among multiple clusters. Specifically, the cluster discrimination unit 37 infers the cluster ID of the target sample. The cluster IDs that can be inferred are limited to the cluster IDs of clusters that can be clustered by the clustering unit 33.

[0092] The anomaly detection unit 38 determines whether a target sample is abnormal based on statistics associated with the cluster identifier to which the target sample belongs in the text information dictionary 36. Specifically, the anomaly detection unit 38 reads the text information dictionary 36 relating to the cluster ID of the target sample and calculates the degree of abnormality of the target text based on statistics associated with all or part of the target text in the read text information dictionary 36. The anomaly detection unit 38 then determines that the target sample is abnormal if the degree of abnormality is greater than a threshold, and determines that the target sample is normal if the degree of abnormality is less than the threshold.

[0093] Alternatively, instead of clustering based on features as described above, clustering based on metadata such as camera location information may be performed. Specifically, the acquisition unit 31 acquires metadata such as the camera location information of the camera that took the photo for each training sample and target sample. The clustering unit 33 divides multiple training samples into multiple clusters based on the metadata of the training samples. The cluster discrimination unit 37 determines which cluster a target sample belongs to from among the multiple clusters based on the metadata of the target sample. By using metadata, it becomes possible to place training samples with similar camera location information into the same cluster, and training samples with different camera location information into different clusters.

[0094] According to the fourth embodiment, it becomes possible to limit the search target of statistics associated with the target text to the statistics of training texts belonging to the same cluster. In other words, it becomes possible to narrow down the search target of statistics associated with the target text to the statistics of training texts corresponding to training samples acquired in an acquisition environment that is somewhat similar to the target text. Therefore, it is expected that the accuracy of anomaly detection will improve.

[0095] <Fifth Embodiment> The anomaly detection device according to the fifth embodiment trains the text generation model used in the text generation unit. The anomaly detection device according to the fifth embodiment will be described below. In this description, parts that are the same as those in the first embodiment described above will be omitted or simplified.

[0096] Figure 14 is a functional block diagram showing an example of an anomaly detection device 40 according to the fifth embodiment. As shown in Figure 14, the anomaly detection device 40 includes an acquisition unit 41, a training unit 42, a text generation model 43, a text generation unit 44, a statistics calculation unit 45, a storage unit 46, a text information dictionary 47, an anomaly detection unit 48, and an output unit 49. The text generation unit 44, statistics calculation unit 45, storage unit 46, anomaly detection unit 48, and output unit 49 are substantially the same as those of the text generation unit 12, statistics calculation unit 14, storage unit 15, anomaly detection unit 17, and output unit 18 according to the first embodiment, respectively.

[0097] The acquisition unit 41 outputs a pair of a sample and text representing the content of that sample. The text prepared is tailored to the anomaly to be detected. For example, if we want to detect a fallen bicycle as an anomaly, but the text generation unit 44 cannot distinguish between fallen and upright bicycles and only outputs the word "bicycle", then a training sample of a fallen bicycle and text containing the phrase "fallen bicycle" are prepared. By training with such text, the text generation unit 44 will be able to output text that can distinguish between fallen and upright bicycles, and it is believed that anomaly detection will be possible even in such situations. In addition, the text may be prepared in a way that does not contain information unnecessary for anomaly detection. For example, text may be prepared that omits weather-related words such as "sunny" or abstract words such as "beautiful".

[0098] The training unit 42 generates a text generation model 43 by training an untrained model to take a sample as input and output text representing the content of that sample, based on the sample and text output by the acquisition unit 41. The untrained model may already be trained on some dataset, and the text generation model 43 may be generated by fine-tuning it. The text generation model 43 is used by the text generation unit 44.

[0099] According to the fifth embodiment, since the text generation model 43 is generated using text that can distinguish the anomaly to be detected, the tendency of the text generated by the text generation unit 44 can be manipulated, and consequently, training text and target text suitable for the anomaly to be detected can be generated. For example, if it is to detect a fallen bicycle as an anomaly, the acquisition unit 41 acquires a sample and text of a fallen bicycle, "fallen bicycle," and a sample and text of a bicycle that is standing, "bicycle that is standing," and the training unit 42 generates a text generation model 43 using these samples and texts. By using the text generation model 43 thus generated, the text generation unit 44 can generate training text and target text that distinguish between fallen bicycles and bicycles that are standing. Therefore, for example, it becomes possible to determine that a target sample including a fallen bicycle is an anomaly, or that a target sample including a bicycle that is standing is normal.

[0100] <Sixth Embodiment> The anomaly detection device according to the sixth embodiment estimates the location of the anomaly within the target sample when the target sample is determined to be abnormal. The anomaly detection device according to the sixth embodiment will be described below. The sample according to the sixth embodiment is assumed to be an image. In addition, in the description of this embodiment, parts that are the same as those in the first embodiment described above will be omitted or simplified.

[0101] Figure 15 is a functional block diagram showing an example of an anomaly detection device 50 according to the sixth embodiment. The anomaly detection device 50 includes an acquisition unit 51, a text generation unit 52, a preprocessing unit 53, a statistical calculation unit 54, a storage unit 55, a text information dictionary 56, an anomaly detection unit 57, an estimation unit 58, and an output unit 59. The acquisition unit 51, text generation unit 52, preprocessing unit 53, statistical calculation unit 54, storage unit 55, and anomaly detection unit 57 are substantially the same as the acquisition unit 11, text generation unit 12, preprocessing unit 13, statistical calculation unit 14, storage unit 15, and anomaly detection unit 17 according to the first embodiment, respectively.

[0102] The estimation unit 58 estimates the image regions (hereinafter referred to as "abnormal regions") corresponding to words in the target image whose statistics show abnormalities (hereinafter referred to as "abnormal words"). The target image with the abnormal regions highlighted is called the abnormal region-enhanced image. Specifically, the estimation unit 58 identifies abnormal words by applying a threshold to the word abnormality score calculated by the anomaly detection unit 57. Next, the estimation unit 58 identifies the abnormal regions corresponding to the abnormal words. As an example, the estimation unit 58 estimates the abnormal regions based on gradient information related to abnormal words in the text generation model. Specifically, it is possible to use a method that identifies regions of interest using gradients such as Guided Back Propagation. As another example, the estimation unit 58 may estimate the abnormal regions by performing object detection using the abnormal words as prompts. Specifically, it is possible to use zero-shot object detection such as GroundingDINO described in Non-Patent Literature 2.

[0103] Figure 16 is a diagram showing the processing flow when an anomaly is detected by the anomaly detection device 50 according to the sixth embodiment. Note that the processing flow during learning by the anomaly detection device 50 according to the sixth embodiment is the same as in the first embodiment and is therefore omitted. Figure 17 is a diagram illustrating one target image I61, which is an example of an inference dataset according to the sixth embodiment.

[0104] Steps S51 to S55 shown in Figure 16 are the same as steps S21 to S25 shown in Figure 4. The target image I61 shown in Figure 17 shows grass and an abnormal object, a bucket, and the target text based on the target image I61 contains the words "grass" and "bucket". Then, in step S55, the word abnormality scores of the words "grass" and "bucket" are calculated. Assume that the word abnormality score of the word "grass" is 0.1 and the word abnormality score of the word "bucket" is 0.9.

[0105] When step S55 is performed, the estimation unit 58 performs an abnormality determination for each word based on the word abnormality score calculated in step S55 (step S56). Specifically, the estimation unit 58 applies a pre-set threshold to the word abnormality score to determine whether each word is abnormal or not. If the word abnormality score is greater than the threshold, the estimation unit 58 determines that the word is abnormal, and if the word abnormality score is less than the threshold, the word is determined to be normal. In this example, the threshold is set to 0.5. The word "grass" has a word abnormality score of 0.1, which is less than the threshold, so it is determined to be normal, and the word "bucket" has a word abnormality score of 0.9, which is greater than the threshold, so it is determined to be abnormal.

[0106] When step S56 is performed, the estimation unit 58 generates an abnormal area visualization image based on the determination result of step S56 (step S57). Specifically, the estimation unit 58 performs zero-shot object detection using the word (abnormal word) that was determined to be abnormal in step S56. Here, it is assumed that GroundingDINO from Non-Patent Literature 2 is used. The estimation unit 58 performs zero-shot object detection using the abnormal word as a prompt, detects the image region (abnormal region) corresponding to the abnormal word, and outputs the target image in which a rectangle surrounding the detected abnormal region is drawn as an abnormal area visualization image. The abnormal area visualization image is an example of an abnormal area enhancement image because the abnormal region is emphasized with a rectangle.

[0107] Figure 18 shows an example of zero-shot object detection results, illustrating an anomaly visualization image I62 in which a rectangle I63 surrounding the anomaly region is drawn. The anomaly visualization image I62 is based on the target image I61 shown in Figure 17. The inference unit 58 performs zero-shot object detection using the anomaly word "bucket" as a prompt, and detects the set of pixels constituting a bucket corresponding to the word "bucket" as an anomaly region. The inference unit 58 draws a rectangle I63 surrounding the anomaly region on the target image I61. In Figure 18, x1 and y1 represent the x and y coordinates of the top-left point of rectangle I63, respectively, and x2 and y2 represent the x and y coordinates of the bottom-right point of rectangle I63, respectively. By drawing rectangle I63 on the target image I61, the anomaly visualization image I62 is generated.

[0108] When step S57 is performed, the output unit 59 outputs the result of determining whether or not there is an anomaly for each word output in step S56 and the anomaly visualization image generated in step S57 (step S58). As an example, the output unit 59 outputs a display screen including the result of determining whether or not there is an anomaly for each word and the anomaly visualization image to a display device.

[0109] Figure 19 illustrates a display screen I7 that includes a display area I72 for determining whether each word has an anomaly and an image I71 for visualizing the anomaly. As shown in Figure 19, the display screen I7 shows the image I71 for visualizing the anomaly and the determination result I72. The image I71 for visualizing the anomaly shown in Figure 19 corresponds to the target image I61 shown in Figure 17. As shown in Figure 19, the image I71 for visualizing the anomaly shows grass and a bucket. As described above, the bucket is detected as an anomaly, so a rectangle I73 surrounding the bucket is drawn.

[0110] Display area I72 shows the determination result I74 of whether or not each word is abnormal, which was output in step S56. The determination result I74 displays the text generated in step S52, the occurrence probability obtained in step S54, the word abnormality score calculated in step S55, and the determination result output in step S56 for the target image I61. A threshold value such as "0.5" may also be displayed alongside the determination result.

[0111] For text, words for which occurrence probability and word abnormality score are to be obtained should be highlighted, for example, with an underline. Specifically, for target image I61, the words "grass" and "bucket" should be highlighted with an underline, while words such as "There" and "are" should not be highlighted because they are not included in the data collection. The occurrence probability, word abnormality score, and judgment result should be displayed aligned below the corresponding word. For example, for the word "grass," the occurrence probability "0.9," word abnormality score "0.1," and judgment result "Normal" are displayed, while for the word "bucket," the occurrence probability "0.1," word abnormality score "0.9," and judgment result "Abnormal" are displayed. Words such as "There" and "are" are not included in the data collection for occurrence probability, word abnormality score, and judgment result, so none of these indicators are displayed.

[0112] Furthermore, the word corresponding to the object may be displayed alongside the object shown in the anomaly visualization image I71. Specifically, the word "bucket" may be displayed below the bucket. This makes it easy for the user to understand the correspondence between the word displayed in the text and the object shown in the anomaly visualization image I71. The output unit 59 may output an anomaly score map instead of the anomaly visualization image, or in parallel with the anomaly visualization image. The anomaly score map can be generated by the estimation unit 58. Specifically, the estimation unit 58 calculates an anomaly score for each pixel, representing the probability that it is an image region (anomaly region) corresponding to an anomaly word, and generates a grayscale image by assigning brightness to the pixel according to the calculated anomaly score. The output unit 59 outputs the generated grayscale image as an anomaly score map. For example, the output unit 59 may display the anomaly score map instead of or alongside the anomaly visualization image I71 shown in Figure 19. In the anomaly score map, the anomaly region is displayed with high brightness, and other regions are displayed with low brightness, resulting in an image that emphasizes the anomaly region. The anomaly score map is an example of an image highlighting anomalies.

[0113] With the above steps completed, the processing by the abnormality detection device 50 according to the sixth embodiment when an abnormality is detected is finished.

[0114] As described above, by displaying an anomaly-highlighted image in which the anomaly area corresponding to the anomaly word is highlighted with a rectangle, users can clearly understand which part of the image was determined to be anomaly. Furthermore, by displaying the anomaly-highlighted image and the judgment result side by side, users can understand the relationship between the anomaly-highlighted image and the judgment result. By displaying the text, occurrence probability, word anomaly degree, and judgment result side by side as the basis for the judgment result, users can understand what part of the text was used to determine whether it was anomaly or normal, and can also assess the accuracy of the judgment result.

[0115] <Seventh Embodiment> The anomaly detection device according to the seventh embodiment edits a text information dictionary. The anomaly detection device according to the seventh embodiment will be described below. In the description of this embodiment, parts that are the same as those of the first embodiment described above will be omitted or simplified.

[0116] Figure 20 is a functional block diagram showing an example of an anomaly detection device 60 according to the seventh embodiment. As shown in Figure 20, the anomaly detection device 60 includes an acquisition unit 61, a text generation unit 62, a statistical calculation unit 63, a storage unit 64, a text information dictionary 65, an anomaly detection unit 66, an output unit 67, an operation input unit 68, and an editing unit 69. The acquisition unit 61, text generation unit 62, statistical calculation unit 63, storage unit 64, anomaly detection unit 66, and output unit 67 are substantially the same as the acquisition unit 11, text generation unit 12, statistical calculation unit 14, storage unit 15, anomaly detection unit 17, and output unit 18 according to the first embodiment, respectively.

[0117] The operation input unit 68 inputs text and / or statistical information about the text according to the user's instructions. Specifically, the operation input unit 68 inputs text to be added to, deleted, or modified in the text information dictionary 65, along with its probability of occurrence. For example, if the word "dog," which is normally normal, is inferred as an anomaly during the initial anomaly detection, the information (dog, p=1.0) is input to the text information dictionary 65 to indicate that the word "dog" appears with a high probability in order to suppress false positives. Another example is if the object "kitchen knife" is to always be detected as an anomaly, the information (kitchen knife, p=0.0) is input. Also, if a word is to be deleted from the text information dictionary, information such as (kitchen knife, p="delete") is input.

[0118] The editing unit 69 edits the text information dictionary 65 based on the information entered by the operation input unit 68. Specifically, the editing unit 69 stores pairs of text and their occurrence probabilities entered by the operation input unit 68 in the text information dictionary 65. If (kitchen knife, p=0.0) is entered by the operation input unit 68 and information such as (kitchen knife, p=0.3) is already registered in the text information dictionary 65, the editing unit 69 may overwrite the already registered information. Also, if (kitchen knife, p="delete") is entered, the editing unit 69 deletes "kitchen knife" and its occurrence probability from the text information dictionary 65. Subsequent anomaly detection is performed according to the edited text information dictionary 65.

[0119] According to the seventh embodiment, a user who has confirmed the result of determining whether or not there is an abnormality can edit the text information dictionary 65 so as not to detect a certain object or to detect a certain object.

[0120] <Eighth Embodiment> The anomaly detection device according to the eighth embodiment calculates word anomaly degree (object appearance anomaly degree) and object disappearance anomaly degree in the anomaly detection unit, calculates image anomaly degree based on the object appearance anomaly degree and object disappearance degree, and determines whether or not there is an anomaly in the target image based on the image anomaly degree. The anomaly detection device according to the eighth embodiment will be described below. In the description of this embodiment, parts that are the same as those in the first embodiment described above will be omitted or simplified.

[0121] The anomaly detection unit 17 calculates the degree of anomaly in object appearance based on the occurrence probability associated with words or combinations of words belonging to specific parts of speech included in the target text in the text information dictionary 16, and calculates the degree of anomaly in object disappearance based on the occurrence probability associated with words or combinations of words stored in the text information dictionary 16 that are not included in the target text, and determines whether or not there is an anomaly in the target sample based on the degree of anomaly in object appearance and the degree of anomaly in object disappearance.

[0122] Specifically, the anomaly detection unit 17, by referring to the text information dictionary 16, calculates the word anomaly degree for each word included in the target text, similar to the first embodiment, and identifies the maximum value among them. This maximum value is defined as the object appearance anomaly degree. Furthermore, the anomaly detection unit 17 extracts words from the text information dictionary 16 that are not included in the target text, and identifies the word with the highest probability of appearance among the extracted words as the word corresponding to the missing object (hereinafter referred to as the missing object word). A missing object is an object that appears with a high probability in the training dataset, but does not appear in the target image. In other words, the absence of a missing object in the target image is highly likely to be an anomaly. Furthermore, the anomaly detection unit 17 identifies the probability of appearance of the missing object word in the text information dictionary 16 as the object disappearance anomaly degree. As another example, the anomaly detection unit 17 may calculate the object disappearance anomaly by applying the probability of appearance of the missing object word in the text information dictionary to an arbitrary function and obtaining a function value. Examples of functions include a function that outputs 0 when it falls below an arbitrary threshold, or an exponential function. The anomaly detection unit 17 then calculates the image anomaly score by weighting the object appearance anomaly score and the object disappearance anomaly score with an arbitrary parameter.

[0123] Figure 21 is a diagram illustrating the processing flow when an anomaly is detected by the anomaly detection device 10 according to the eighth embodiment. Figure 22 is a diagram illustrating four normal images I81 to I84, which are an example of a training dataset according to the eighth embodiment. The processing flow during training by the anomaly detection device 10 according to the eighth embodiment is the same as in the first embodiment and is therefore omitted. As a result of training based on the training dataset shown in Figure 22, the probability of the word "bucket" appearing is 1, the probability of the word "flower" appearing is 3 / 4, the probability of the word "grass" appearing is 3 / 4, and the probability of the word "butterfly" appearing is 1 / 4. Figure 23 is a diagram illustrating target image I91 included in the inference dataset. Target image I91 shows a flower and a butterfly, but the bucket, which was shown in all four normal images I81 to I84, is not shown. Since the bucket is not shown in target image I91, it is expected to be judged as an anomaly.

[0124] Steps S61-S63 shown in Figure 21 are the same as steps S21-S23 shown in Figure 4. The target text representing the content of the target image includes the words "flower" and "butterfly".

[0125] When step S63 is performed, the anomaly detection unit 17 calculates the object appearance anomaly degree for the words that appear in the target text (step S64). Specifically, the anomaly detection unit 17 refers to the text information dictionary 16, as in the first embodiment, to identify the word anomaly degree for the words that appear in the target text, and the maximum value among the identified word anomaly degrees is set as the object appearance anomaly degree. In the case of the target image I91 shown in Figure 23, the word anomaly degree for the word "flower" is 0, and the word anomaly degree for the word "butterfly" is 3 / 4. Since the maximum value of these word anomaly degrees is 3 / 4, the object appearance anomaly degree is 3 / 4.

[0126] When step S64 is performed, the anomaly detection unit 17 calculates the object disappearance anomaly degree for words that do not appear in the target text (step S65). Specifically, the anomaly detection unit 17 extracts words that are registered in the text information dictionary 16 but do not appear in the target text (missing object words). In the case of the target image I91, the words "bucket" and "grass" are missing object words. The occurrence probabilities of the words "bucket" and "grass" stored in the text information dictionary 16 are 1 and 3 / 4, respectively, and the maximum value of these probabilities, 1, is set as the object disappearance anomaly degree.

[0127] When step S65 is performed, the anomaly detection unit 17 calculates the image anomaly score based on the object appearance anomaly score calculated in step S65 and the object disappearance anomaly score calculated in step S66 (step S66). Specifically, the anomaly detection unit 17 calculates a weighted average of the object appearance anomaly score and the object disappearance anomaly score. More specifically, the anomaly detection unit 17 calculates the image anomaly score using the formula Image Anomaly Score = α × Object Appearance Anomaly Score + (1 - α) × Object Disappearance Anomaly Score, using a parameter α set in advance by the user. If α = 1 / 2, the anomaly score of the target image I91 is (1 / 2) × (3 / 4) + (1 / 2) × 1 = 7 / 8.

[0128] In steps S67 to S68, the presence or absence of abnormalities in the target image is determined based on the image abnormality score, and the determination result is output. Steps S67 to S68 are the same as steps S27 to S28 shown in Figure 4, so their explanation is omitted.

[0129] With the above steps completed, the processing by the abnormality detection device 10 according to the eighth embodiment when an abnormality is detected is finished.

[0130] In the examples shown in Figures 22 and 23, in the first embodiment, the object appearance anomaly score is the image anomaly score, so the image anomaly score is 3 / 4. This is because a butterfly, which does not appear often in the training dataset, is captured in the target image I91. On the other hand, in the eighth embodiment, the image anomaly score is 7 / 8, which is greater than the image anomaly score of 3 / 4 in the first embodiment. This is because, in addition to a butterfly, which does not appear often in the training dataset, being captured in the target image I91, a bucket, which appears in all of the training dataset, is not captured in the target image I91. Therefore, according to the eighth embodiment, it becomes possible to determine the presence or absence of anomalies by taking into account not only the appearance of abnormal objects but also the disappearance of objects that should appear.

[0131] <Ninth Embodiment> The anomaly detection device according to the ninth embodiment acquires video as a sample and calculates the degree of anomaly on a video-by-video basis. The anomaly detection device according to the ninth embodiment will be described below. In the description of this embodiment, parts that are the same as those of the first embodiment described above will be omitted or simplified.

[0132] Figure 24 is a functional block diagram showing an example of an anomaly detection device 70 according to the ninth embodiment. As shown in Figure 24, the anomaly detection device 70 includes an acquisition unit 71, a text generation unit 72, a preprocessing unit 73, an integration unit 74, a statistical calculation unit 75, a storage unit 76, a text information dictionary 77, an anomaly detection unit 78, and an output unit 79. The storage unit 76 and the output unit 79 are substantially the same as the storage unit 15 and the output unit 18 according to the first embodiment, respectively.

[0133] The acquisition unit 71 acquires video as a sample. During training, the acquisition unit 71 acquires a training dataset containing multiple videos. The training dataset contains multiple normal videos. When an anomaly is detected, the acquisition unit 71 acquires an inference dataset containing one or more videos. A video is a single dataset containing multiple time-series frames.

[0134] The text generation unit 72 generates multiple texts corresponding to multiple frames contained in the video. The method of generating the text is the same as in the first embodiment. Alternatively, instead of generating text for all frames of the video, text may be generated every few frames, etc. This operation outputs multiple text sets from a single video. The text generation unit 72 performs processing on both the training video and the target video.

[0135] The preprocessing unit 73 performs word segmentation on the text contained in the text group of each video. Similar to the first embodiment, other preprocessing such as part-of-speech determination and stemming may also be performed. This operation yields a word sequence equal to the number of processed frames for each video. The preprocessing unit 73 performs processing on both the training video and the target video.

[0136] The integration unit 74 integrates multiple texts related to one video into a single text that represents the content of that video. Specifically, for each of the multiple frames contained in one video, the integration unit 74 generates a word sequence without duplication by selecting words that appear at least once in the video from words belonging to a specific part of speech contained in the target text. Integration is performed by taking the logical OR of all the elements of the word sequence. As a specific example, the word sequence (cat, dog) obtained from the first frame of the first video and the word sequence (person, dog) obtained from the second frame of the first video are integrated to generate the word sequence (person, cat, dog). This operation yields one word sequence for each video. The word sequence obtained by the integration unit 74 is considered to be the word sequence corresponding to one video, and the subsequent processing is performed in the same manner as in the first embodiment to calculate the abnormality level of each video.

[0137] The statistics calculation unit 75 calculates statistics for each word included in the word sequence generated by the integration unit 74. The anomaly detection unit 78 determines whether or not there is an anomaly in the target video, which is the target sample, based on the statistics associated with each word in the text information dictionary 77.

[0138] Figure 25 is a diagram showing the processing flow during learning by the anomaly detection device 70 according to the ninth embodiment. Figure 26 is a diagram illustrating three normal videos M1, M2, and M3, which are an example of a learning dataset according to the ninth embodiment. Each normal video M1, M2, and M3 contains three frames Fij (i is an index representing the video number to which the frame belongs; i=1,2,3. j is an index representing the frame number; j=1,2,3). Each normal video M1, M2, and M3 is captured by a moving camera. The text shown at the top of each frame Fij is the caption generated in step S73.

[0139] Step S71 is the same as step S11 shown in Figure 2. Once step S71 is performed, the acquisition unit 71 acquires a normal image (step S72). First, let's assume that the normal image M1 shown in Figure 26 has been acquired.

[0140] When step S72 is performed, the text generation unit 72 generates text representing the content of each of the three frames included in the normal video acquired in step S72 (step S73). For example, by using a caption generation model, a caption is generated that represents the content of each frame in sentence form. As a result, as shown in Figure 26, the following are generated for frame F11 of the normal video M1: "There are a flower and grass.", for frame F12: "There is a bicycle.", and for frame F13: "There are a flower and grass."

[0141] When step S73 is performed, the preprocessing unit 73 performs preprocessing on the text generated in step S73 (step S74). Specifically, the preprocessing unit 73 performs word segmentation on the text and further extracts words that belong to nouns. As a result, the word sequences (flower, grass) are obtained from frame F11, (bicycle) from frame F12, and (flower, grass) from frame F13.

[0142] When step S74 is performed, the integration unit 74 integrates the word sequences of each frame output in step S74 so that there are no duplicates for each normal video (step S75). Specifically, the integration unit 74 integrates the three word sequences (flower, grass), (bicycle), and (flower, grass) so that there are no duplicate words, and generates the word sequence (flower, grass, bicycle).

[0143] Steps S76 to S79 ​​are the same as steps S15 to S18 shown in Figure 2 and are therefore omitted. However, in step S78, the statistics calculation unit 75 calculates the probability of occurrence of each word by dividing the value of the word counter for each word by the number of normal images included in the training dataset. As a result, the probability of occurrence of the word "flower" is 1, the probability of occurrence of the word "grass" is 1, the probability of occurrence of the word "bicycle" is 1, the probability of occurrence of the word "wall" is 1 / 3, and the probability of occurrence of the word "butterfly" is 1 / 3.

[0144] With the above steps completed, the learning process by the anomaly detection device 70 according to the ninth embodiment is finished.

[0145] Figure 27 is a diagram showing the processing flow when an anomaly is detected by the anomaly detection device 70 according to the ninth embodiment. Figure 28 is a diagram illustrating one target video M4, which is an example of an inference dataset according to the ninth embodiment. The target video M4 is assumed to be an anomaly video because the bicycle that should be visible is not visible.

[0146] Steps S81 to S83 are the same as steps S21 to S23 in Figure 4. In the target video M4 shown in Figure 28, the first frame F41 shows grass and flowers, the second frame F43 shows a butterfly, and the third frame F43 shows grass and flowers. The target text representing the content of the first frame F41 contains the words "grass" and "flower," the target text representing the content of the second frame F42 contains the word "butterfly," and the target text representing the content of the third frame F43 contains the words "grass" and "flower."

[0147] When step S83 is performed, the integrating unit 74 integrates the words of each frame F41, F42, and F43 so that there are no duplicates (step S84). This results in the word sequence (flower, grass, butterfly).

[0148] When step S84 is performed, the anomaly detection unit 78 calculates the object appearance anomaly degree for the words included in the word sequence after integration in step S84, in other words, for the words that appeared in the text of the normal video (step S85). The method for calculating the object appearance anomaly degree is the same as in the eighth embodiment, so the explanation is omitted. In step S85, an object appearance anomaly degree of 2 / 3 is obtained for the word "butterfly".

[0149] When step S86 is performed, the anomaly detection unit 78 calculates the object disappearance anomaly degree for words that are not included in the word sequence after integration in step S84, in other words, words that do not appear in the text of the normal video (step S86). The method for calculating the object disappearance anomaly degree is the same as in the eighth embodiment, so the explanation is omitted. In step S86, an object disappearance anomaly degree of 1 is obtained for the word "bicycle".

[0150] When step S86 is performed, the anomaly detection unit 78 calculates the image anomaly score based on the object appearance anomaly score calculated in step S85 and the object disappearance anomaly score calculated in step S86 (step S87). The method for calculating the image anomaly score is the same as in the eighth embodiment, so the explanation is omitted. Specifically, when a weighted average is calculated with α=0.5, the anomaly score of the normal image M4 is 1 / 2×2 / 3+1 / 2×1=5 / 6.

[0151] Steps S88 to S89 are the same as steps S27 to S28 shown in Figure 4, so their explanation will be omitted.

[0152] With the above steps completed, the processing by the abnormality detection device 70 according to the ninth embodiment when an abnormality is detected is finished.

[0153] According to the ninth embodiment, it becomes possible to determine whether or not there are abnormalities in the target video. Furthermore, by considering the degree of abnormality in object appearance and the degree of abnormality in object disappearance, it becomes possible to detect the appearance of abnormal objects and the disappearance of normal objects.

[0154] <Tenth Embodiment> The anomaly detection device according to the 10th embodiment uses a feature extraction unit instead of a statistics calculation unit and an anomaly detection model instead of a text information dictionary. The anomaly detection device according to the 10th embodiment will be described below. In the description of this embodiment, parts that are the same as those of the first embodiment described above will be omitted or simplified.

[0155] Figure 29 is a functional block diagram showing an example of an anomaly detection device 80 according to the tenth embodiment. As shown in Figure 29, the anomaly detection device 80 includes an acquisition unit 81, a text generation unit 82, a feature extraction unit 83, a training unit 84, an anomaly detection model 85, an anomaly detection unit 86, and an output unit 87. The acquisition unit 81, the text generation unit 82, and the output unit 87 are substantially the same as the acquisition unit 11, the text generation unit 12, and the output unit 18 according to the first embodiment, respectively.

[0156] The feature extraction unit 83 extracts features related to the target text based on the target text generated by the text generation unit 82. The feature extraction unit 83 also extracts features related to the training text based on the training text generated by the text generation unit 82. Feature vectors are used as the features. While it is assumed that a text embedding model using a Transformer is used for the conversion to feature vectors, techniques such as Bag of Words or Doc2Vec may also be used.

[0157] The training unit 84 trains an untrained machine learning model based on the features of the training text extracted by the feature extraction unit 83, and generates an anomaly detection model that takes a sample as input and detects anomalies in that sample. If the training dataset includes both anomaly and normal samples and the samples are annotated with labels, a supervised classification model such as a neural network or a support vector machine may be used as the anomaly detection model. Alternatively, if the training dataset consists only of normal samples, a model that determines anomalies based on the distance to neighboring samples in the feature space may be used. In this case, the training unit 84 stores the features of the training dataset as a model. Furthermore, a network that approximates a Gaussian distribution of normal features may be trained, as in the method using Normalizing flow described in Non-Patent Document 3.

[0158] The anomaly detection unit 86 determines whether or not there is an anomaly in the target sample based on the anomaly detection model generated by the training unit 84 and the features related to the target text extracted by the feature extraction unit 83.

[0159] As described above, the anomaly detection device 80 according to the 10th embodiment converts text into features and uses these features to determine whether or not there is an anomaly in the sample. These features have values ​​that reflect not only the objects contained in the sample, but also complex relationships such as co-occurrence relationships between the objects contained in the sample. Therefore, the anomaly detection device 80 can determine whether or not there is an anomaly by taking into account complex relationships such as co-occurrence relationships between the objects contained in the sample, compared to determining the presence or absence of an anomaly using statistical quantities such as word occurrence probabilities.

[0160] <Hardware Configuration> Figure 30 shows the hardware configuration of the anomaly detection devices 10 to 90 according to the first to tenth embodiments described above. In Figure 30, the anomaly detection devices 10 to 90 according to the first to tenth embodiments are collectively referred to as anomaly detection device 100. As shown in Figure 30, the anomaly detection device 100 is a computer having a processor 101, ROM (Read Only Memory) 102, RAM (Random Access Memory) 103, auxiliary storage device 104, input device 105, display device 106, and communication device 107. Data and various signals are transmitted and received between the processor 101, ROM 102, RAM 103, auxiliary storage device 104, input device 105, display device 106, and communication device 107 via a bus.

[0161] The processor 101 is an integrated circuit that controls the overall operation of the anomaly detection device 100. For example, the processor 101 has a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a DSP (Digital Signal Processor), and / or an FPU (Floating-Point Unit). The processor 101 may also have internal memory and I / O interfaces. The processor 101 performs the various processes described above by interpreting and calculating programs pre-stored in the ROM 102 or auxiliary storage device 104, etc. The processor 101 may be partially or entirely implemented by hardware such as an ASIC (Application Specific Integrated Circuit) or FPGA (Field Programmable Gate Array).

[0162] ROM 102 is a non-volatile memory that stores various types of data. For example, ROM 102 stores data and settings used by the processor 101 when it performs various processes. ROM 102 may also have a non-temporary computer-readable storage medium that stores programs executed by the processor 101.

[0163] RAM103 is volatile memory used for reading and writing data. RAM103 temporarily stores data used when the processor 101 performs various processes. RAM103 provides the work area for the processor 101.

[0164] The auxiliary storage device 104 is a non-volatile memory that stores various types of data. For example, the auxiliary storage device 104 stores data and settings used by the processor 101 when it performs various processes, and data generated by various processes performed by the processor 101. The auxiliary storage device 104 is composed of an HDD (Hard Disk Drive), an SSD (Solid State Drive), an integrated circuit storage device, etc. The auxiliary storage device 104 may also have a non-temporary computer-readable storage medium that stores programs executed by the processor 101.

[0165] The input device 105 receives various operation inputs from the operator. The input device 105 can include a keyboard, mouse, various switches, touchpad, touch panel display, etc. Electrical signals corresponding to the received operation inputs are supplied to the processor 101.

[0166] The display device 106 displays various data according to the control of the processor 101. The display device 106 can be a CRT (Cathode-Ray Tube) display, a liquid crystal display, an organic EL (Electro-Luminescence) display, an LED (Light-Emitting Diode) display, a plasma display, or any other display as appropriate. The display device 106 may also be a projector.

[0167] The communication device 107 includes a communication interface such as a network interface card (NIC) for data communication with various devices connected to the anomaly detection device 100 via a network. Electrical signals may be supplied from a computer connected via the communication device 107 or from an input device on that computer, or various data may be displayed on a display device on a computer connected via the communication device 107. The input device 105 can be replaced by a computer connected via the communication device 107 or from an input device on that computer, and the display device 106 can be replaced by a display device on a computer connected via the communication device 107.

[0168] The anomaly detection device 100 does not need to include all of the processor 101, ROM 102, RAM 103, auxiliary storage device 104, input device 105, display device 106, and communication device 107. Some of the ROM 102, RAM 103, auxiliary storage device 104, input device 105, display device 106, and communication device 107 may be omitted as needed. The anomaly detection device 100 may also be equipped with any additional hardware devices useful for executing the processing according to this embodiment. The anomaly detection device 100 does not need to consist of a single physical computer; it may consist of a computer system having multiple computers connected via wired or network lines for communication. The assignment of the series of processes according to this embodiment to the multiple processors 101 implemented in each of the multiple computers can be arbitrarily configured. All processors 101 may execute all processes in parallel, or specific processes may be assigned to one or some of the processors 101, and the series of processes according to this embodiment may be executed by the entire computer system.

[0169] According to the embodiment described above, it is possible to provide an anomaly detection device that is robust to changes in the shooting environment.

[0170] While several embodiments of the present invention have been described, these embodiments are presented as examples only and are not intended to limit the scope of the invention. These novel embodiments can be carried out in a variety of other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their variations are included in the scope and spirit of the invention, as well as in the claims of the invention and its equivalents. [Explanation of Symbols]

[0171] 10...Anomaly detection device, 11...Acquisition unit, 12...Text generation unit, 13...Preprocessing unit, 14...Statistical calculation unit, 15...Storage unit, 16...Text information dictionary, 17...Anomaly detection unit, 18...Output unit.

Claims

1. An acquisition unit that acquires the first sample to be detected as an anomaly, A generation unit that uses a trained model to generate a first text representing the content of the first sample from the first sample, A unit that determines whether or not there is an anomaly in the first sample based on statistical quantities linked to all or part of the first text in a text information dictionary, wherein the text information dictionary links all or part of the second text representing the content of the second sample included in the training dataset with statistical quantities related to the frequency of occurrence of all or part of the second text in the training dataset, and includes a detection unit. An output unit that outputs the result of determining whether or not there is an abnormality in the first sample, An anomaly detection device equipped with the following:

2. The aforementioned trained model is a caption generation model, The generation unit applies the first sample to the caption generation model to generate a caption that describes the content of the first sample as the first text. An anomaly detection device according to claim 1.

3. The aforementioned trained model is a model that takes a sample and a prompt as input and outputs text corresponding to the combination of the sample and the prompt. The acquisition unit further acquires a prompt for the content of the first sample, The generation unit applies the first sample and the prompt to the model to generate the first text. An anomaly detection device according to claim 1.

4. It is further equipped with a statistical calculation unit, The acquisition unit acquires a plurality of second samples included in the training data, The generation unit uses the trained model to generate a second text representing the content of the second sample from each of the plurality of second samples. The statistical calculation unit calculates the statistical amount for all or part of the second text for each of the plurality of second samples. Anomaly detection device according to claim 1

5. The statistical calculation unit calculates the frequency of occurrence of all or part of the second text corresponding to each of the plurality of second samples, and calculates the statistical quantity based on the frequency of occurrence. The detection unit calculates the degree of abnormality of the first text based on the statistical quantity associated with all or part of the first text, and determines that the first sample is abnormal if the degree of abnormality is greater than a threshold, and determines that the first sample is normal if the degree of abnormality is less than the threshold. Anomaly detection device according to claim 4

6. The aforementioned training sample is a normal sample that does not contain any abnormalities. The statistical calculation unit calculates the probability of occurrence based on the frequency of occurrence of the second text as the statistical quantity. The detection unit calculates the degree of abnormality of the first text based on the probability of occurrence associated with the first text. Anomaly detection device according to claim 5

7. An anomaly detection device according to claim 4, further comprising a preprocessing unit that performs preprocessing for dividing the first text and / or the second text into a plurality of parts, and / or preprocessing for removing information unnecessary for anomaly detection from the first text and / or the second text.

8. The statistical calculation unit calculates the statistical amount for words or combinations of words belonging to a specific part of speech contained in the second text. The aforementioned text information dictionary links the aforementioned words or combinations of words with the aforementioned statistics, The detection unit identifies a statistical quantity associated with each word or combination of words belonging to a specific part of speech contained in the first text in the text information dictionary, calculates a word abnormality score based on the identified statistical quantity, determines that the first sample is abnormal if the maximum value of the calculated word abnormality score is greater than a threshold, and determines that the first sample is normal if the maximum value is less than the threshold. An anomaly detection device according to claim 4.

9. It further includes an estimation unit, The first sample and the second sample are images. The estimation unit estimates the image region corresponding to the word in the first sample in which the statistic shows an abnormality. An anomaly detection device according to claim 8.

10. The anomaly detection device according to claim 9, wherein the estimation unit estimates the image region based on gradient information relating to the abnormal word of the trained model.

11. The anomaly detection device according to claim 9, wherein the estimation unit estimates the image region by performing object detection using the abnormal word as a prompt.

12. It further comprises a clustering unit and a cluster determination unit, The clustering unit performs clustering on the training dataset, dividing the multiple second samples into multiple clusters. The statistical calculation unit calculates the statistical quantity for each of the plurality of clusters, The text information dictionary associates all or part of the second text with the statistics for each identifier of the multiple clusters. The cluster determination unit determines the first cluster to which the first sample belongs from among the plurality of clusters. The detection unit determines whether or not there is an abnormality in the first sample based on the statistical quantity associated with the identifier of the first cluster in the text information dictionary. An anomaly detection device according to claim 4.

13. The anomaly detection device according to claim 12, wherein the clustering unit performs the clustering using an unsupervised clustering method.

14. The clustering unit performs the clustering based on the metadata of the second sample. The cluster determination unit determines the cluster to which the first sample belongs based on the metadata of the first sample. An anomaly detection device according to claim 12.

15. It also includes an integration section, The first sample and the second sample are a single dataset containing multiple time-series frames. The generation unit generates multiple texts corresponding to each of the multiple frames, The integration unit integrates the plurality of texts into a single first text that represents the contents of the single dataset. An anomaly detection device according to claim 4.

16. The integration unit generates a sequence of words without duplicates by selecting, for each of the plurality of frames, words belonging to a specific part of speech contained in the first text that appear once or more in the single dataset. The statistic calculation unit calculates the statistic for each word included in the word sequence, The detection unit determines whether or not there is an abnormality in the first sample based on the statistical quantity associated with each word included in the word sequence in the text information dictionary. An anomaly detection device according to claim 15.

17. An input unit for inputting text and / or statistical information about said text according to user instructions, The system further comprises an editorial unit that edits the text information dictionary based on the input information, An anomaly detection device according to claim 1.

18. An anomaly detection device according to claim 1, further comprising: a first training unit that generates a trained model by training an untrained model to take a sample as input and output the text representing the content of the sample.

19. The system further comprises a feature extraction unit that extracts features from all or part of the second text, The aforementioned text information dictionary links the aforementioned statistics, the aforementioned features, and all or part of the second text, The detection unit calculates the degree of abnormality of all or part of the first text based on the statistical quantity and feature quantity associated with all or part of the first text in the text information dictionary, and determines that the first sample is abnormal if the maximum value of the calculated degree of abnormality is greater than a threshold, and determines that the first sample is normal if the maximum value is less than the threshold. An anomaly detection device according to claim 4.

20. The statistical calculation unit calculates the probability of occurrence of a word or combination of words belonging to a specific part of speech contained in the second text, based on the frequency of occurrence of such words or combinations of words, as the statistical quantity. The detection unit calculates the degree of abnormality of a word or combination based on the occurrence probability and feature quantities associated with a word or combination of words belonging to a specific part of speech contained in the first text in the text information dictionary. An anomaly detection device according to claim 19.

21. The statistical calculation unit calculates the probability of occurrence of a word or combination of words belonging to a specific part of speech contained in the second text, based on the frequency of occurrence of such words or combinations of words, as the statistical quantity. The detection unit calculates the degree of abnormality in object appearance based on the occurrence probability associated with a word or combination of words belonging to a specific part of speech included in the first text in the text information dictionary, calculates the degree of abnormality in object disappearance based on the occurrence probability associated with a word or combination of words stored in the text information dictionary that is not included in the first text, and determines whether or not there is an abnormality in the first sample based on the degree of abnormality in object appearance and the degree of abnormality in object disappearance. An anomaly detection device according to claim 4.

22. Instead of the statistical calculation unit, a feature extraction unit is provided that extracts a first feature quantity related to the first text based on the first text and a second feature quantity related to the second text based on the second text. The system further comprises a second training unit that trains an anomaly detection model to detect anomalies in the second sample using the second feature quantity, The detection unit determines whether or not the first sample has an anomaly based on the anomaly detection model and the first feature quantity. An anomaly detection device according to claim 1.

23. The anomaly detection device according to claim 1, wherein the output unit displays the first sample, the first text, and the judgment result side by side on a display device.

24. The output unit is, The first sample, the first text, and the judgment result are displayed side by side on a display device. A specific word in the first text having the maximum value is displayed with different visual effects depending on whether the maximum value is greater than or less than the threshold. An anomaly detection device according to claim 8.

25. The anomaly detection device according to claim 24, wherein the output unit displays the maximum value in conjunction with the specific word.

26. The processor, The acquisition process involves obtaining the first sample to be detected as an anomaly, A generation step of generating a first text representing the content of the first sample from the first sample using a trained model, A unit that determines whether or not there is an abnormality in the first sample based on statistical quantities linked to all or part of the first text in a text information dictionary, wherein the text information dictionary links all or part of the second text representing the content of the second sample included in the training data with statistical quantities related to the frequency of occurrence of all or part of the second text in the training data, a detection step, An output step that outputs the result of determining whether or not there is an abnormality in the first sample, An anomaly detection method comprising the following:

27. In the processor, An acquisition function that obtains the first sample to be detected as an anomaly, A generation function that uses a trained model to generate a first text representing the content of the first sample from the first sample, A function that determines whether or not there is an anomaly in the first sample based on statistical quantities linked to all or part of the first text in a text information dictionary, wherein the text information dictionary has a detection function that links all or part of the second text representing the content of the second sample included in the training data with statistical quantities related to the frequency of occurrence of all or part of the second text in the training data. An output function that outputs the result of determining whether or not there is an abnormality in the first sample, An anomaly detection program that enables this.

Images