Determination system and determination method

Abstract

To reduce a cost for annotating information of a state reflected in a photographed image.SOLUTION: The determination system 100 includes the image classification unit 13 that labels an unknown image similar to a manual abnormal image as an abnormal state and labels an unknown image not similar to the manual abnormal image as a normal state based on the manual abnormal image labeled as an abnormal state and the unknown image whose state is unknown to create learning data for determining a state shown in an image, and the model learning unit 15 that creates the anomaly detection model 16 that outputs a state shown in an image as an input by machine learning using the learning data as an input.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a determination system and a determination method. [Background technology]

[0002] In recent years, various anomaly detection technologies using image analysis have been developed and are being introduced into social infrastructure fields such as manufacturing to improve work efficiency. Among these, methods that apply machine learning are being developed with the advantage that they differ from conventional image analysis technologies in that they can automatically determine the image patterns to be detected.

[0003] For example, Patent Document 1 describes a water quality analyzer having the following features. "A water quality analysis device comprising: a hue analysis unit that learns a convolutional AE (Auto Encoder) on image data of a sedimentation tank to extract hue feature quantities of the sedimentation tank, classifies the image data into hue clusters by clustering from the hue feature quantities, and plots a hue feature space and cluster distribution; and a floating matter analysis unit that learns a convolutional AE on the image data to extract feature quantities of suspended matter in the sedimentation tank, classifies the image data into suspended matter clusters by clustering from the suspended matter feature quantities, and plots a floating matter feature space and cluster distribution." Here, convolution AE is unsupervised learning, and learning is performed using training data as input images, so there is an advantage that the user does not need to label the color and suspended matter in the image of the treated water. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2019-136664 Summary of the Invention [Problem to be solved by the invention]

[0005] In addition to unsupervised learning as in Patent Document 1, machine learning technology also uses supervised learning. Supervised learning requires preliminary "annotation," in which humans manually label the states of the training data. This annotation takes time, and there is a possibility that the annotation criteria will vary from person to person. The more abnormal images labeled with abnormal states through annotation, the more accurate a learning model can be constructed by using those abnormal images as training data for supervised learning. Furthermore, a large number of abnormal images are used not only as training data but also for model verification. However, the high cost of visual annotation by humans makes it difficult to prepare abnormal images.

[0006] Therefore, the main object of the present invention is to reduce the cost of annotating information about the state of a captured image. [Means for solving the problem]

[0007] The determination system of the present invention has the following features. The present invention is characterized by having an image classification unit that creates learning data for determining the state shown in an image by labeling the unknown images similar to the confirmed abnormal images as abnormal states and the unknown images whose states are unknown, based on confirmed abnormal images labeled as showing an abnormal state and labeling the unknown images not similar to the confirmed abnormal images as normal states. Other means will be described later. [Effects of the Invention]

[0008] According to the present invention, it is possible to reduce the cost of annotating information about the state shown in a captured image. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a plan view of a painting wastewater treatment plant according to this embodiment. [Figure 2] FIG. 1 is a configuration diagram of a determination system according to an embodiment of the present invention. [Figure 3] FIG. 10 is an explanatory diagram showing a classification process of captured images according to the present embodiment. [Figure 4] 1 is a flowchart showing an outline of processing performed by a learning device according to an embodiment of the present invention. [Figure 5] 5 is a flowchart showing details of a process for classifying the feature amounts of each image in FIG. 4 into provisional classes according to this embodiment. [Figure 6] FIG. 10 is an explanatory diagram showing the process of clustering processing in an image feature space according to the present embodiment. [Figure 7] FIG. 2 is an explanatory diagram showing an example of a captured image according to the present embodiment. [Figure 8] 8 is a graph showing the results of analysis by a model learning unit of the captured image of FIG. 7 according to this embodiment. [Figure 9] FIG. 2 is a hardware configuration diagram of each device of the determination system according to the present embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0011] FIG. 1 is a plan view of the painting wastewater treatment plant. When painting wastewater generated at a manufacturing site 211 flows into the painting wastewater treatment plant, it purifies the wastewater using chemicals and discharges the treated wastewater into an outside location 212, such as the sea or a river. In Figure 1, the flow of wastewater is indicated by wavy arrows. Specifically, wastewater treatment is carried out in the following manner. (Step 1) Wastewater generated at a manufacturing site 211 is treated in a pH adjustment tank 201 and an oxidation tank 202. For example, the pH of the wastewater is adjusted by a chemical introduced through a pH adjuster introduction pipe 221. (Step 2) In the condensation tank 203, the sludge is condensed by a chemical introduced through the condensation agent introduction pipe 222. (Step 3) The sludge is settled in the settling tank 204 and separated from the wastewater. (Step 4) After stirring in the settling tank 204, if no sludge floats to the surface of the water, the water is discharged outside 212.

[0012] Here, workers check the level of condensation and floating matter, and if any abnormalities are found, they take action on-site, which is a manual task. Workers check each piece of equipment in the painting wastewater treatment plant by walking along the catwalk 205, which is a walkway. Furthermore, instead of having a person visually check the state of the wastewater, the state of the wastewater may be automatically checked by photographing the wastewater and machine-based image recognition. For this reason, the painting wastewater treatment plant is equipped with a camera 223 that photographs the wastewater in the condensation tank 203 and a camera 224 that photographs the wastewater in the sedimentation tank 204.

[0013] FIG. 2 is a diagram showing the configuration of the determination system 100. The determination system 100 is applied to, for example, the painting wastewater treatment plant shown in FIG. 1, and automatically recognizes the state of the wastewater from images captured by the cameras 223 and 224. Therefore, the judgment system 100 has a monitoring device (monitoring unit) 2 that monitors images captured by the monitoring camera 20, and a learning device 1 that uses machine learning to create a model (anomaly detection model 16) that is referenced to perform the monitoring process (drainage status judgment process) of the monitoring device 2.

[0014] For ease of explanation, Fig. 2 shows separate monitoring camera 20 that inputs captured images to monitoring device 2 and learning camera 10 that inputs captured images to learning device 1. However, these two cameras may be physically the same IP camera (camera 223 or camera 224 in Fig. 1), with the IP camera operating as monitoring camera 20 during the monitoring period and operating as learning camera 10 during the learning period (before the monitoring period). Furthermore, the components of learning device 1 and monitoring device 2 may be housed in the same housing. Learning device 1 and monitoring device 2 may each be a resource-saving computing device similar to a general embedded computer, or a resource-rich computing device such as a desktop personal computer.

[0015] FIG. 3 is an explanatory diagram showing the classification process of the captured images. The learning device 1 classifies the images captured by the learning camera 10 as follows. All captured images acquired from the learning camera 10 are referred to as "acquired images" (S11). If manual annotation (S12) is performed by the field worker 24 among the acquired images (Yes), the images are classified as "manual normal images" (S13) determined to be in a normal state or "manual abnormal images (confirmed abnormal images)" (S14) determined to be in an abnormal state.

[0016] Annotation is a procedure in which the on-site worker 24 determines the state of the drainage water by visually checking the state of the drainage water shown in the image or by checking the equipment operation records. Note that the manual annotation (S12) may include not only the input of image labels by the on-site worker 24 (user) but also input in which the user's input is supported by the generation AI (input based on the user's input).

[0017] On the other hand, acquired images for which manual annotation (S12) has not been performed (No) are classified as unknown images (S15). The unknown images are automatically classified (S16) by the image classification unit 13, and are classified as "automatic normal images" (S17) determined to be in a normal state or "automatic abnormal images" (S18) determined to be in an abnormal state. In other words, based on abnormal manual images labeled as showing an abnormal state and unknown images whose state is unknown, the image classification unit 13 creates learning data for determining the state shown in the image by labeling unknown images similar to the abnormal manual images as an abnormal state and labeling unknown images not similar to the abnormal manual images as a normal state.

[0018] Returning to FIG. 2, the learning device 1 includes an image acquisition unit 11, an image preprocessing unit 12, an image classification unit 13, a classification result storage unit 14, a model learning unit 15, and an anomaly detection model 16. The image acquisition unit 11 acquires, as acquired images, images captured by the connected learning camera 10. For ease of explanation, the acquired images are classified into two types of states: a normal state and an abnormal state, but multiple abnormal states (such as a mildly abnormal state and a severely abnormal state) may also be distinguished. In this case, the anomaly detection model 16 can output multiple types of abnormal conditions in a distinguishable manner, and the monitoring device 2 notifies the on-site worker 24 at the wastewater treatment plant of the multiple types of abnormal conditions shown in the judgment image output from the anomaly detection model 16 in a distinguishable manner.

[0019] The image preprocessing unit 12 performs the following preprocessing on the images acquired by the image acquisition unit 11 so that the image classification unit 13 can perform classification processing. A part of the acquired images (for example, about five images) is checked by the field worker 24, and the images are annotated (manually labeled) as either normal or abnormal. On the other hand, the remaining acquired images that have not been annotated are unknown images. Extract image features (such as luminance value statistics) from manual abnormal images and unknown images to be used as input data for classification processing in the image classification unit 13.

[0020] The image classification unit 13 generates a class to which each image belongs by using a clustering method (e.g., K-means method) for an image feature space in which the image features of each image extracted by the image preprocessing unit 12 are arranged, and determines whether the unknown image is similar to a manual abnormal image depending on the proportion of manual abnormal images included in the generated class. The K-means method is a classification algorithm based on the correlation between input data. The K-means method classifies unknown images that are similar to manual abnormal images into automatic abnormal images, and classifies unknown images that are not similar to manual abnormal images into automatic normal images.

[0021] As the initial state of the K-means method, the image classification unit 13 prepares an image feature space in which a random class (two classes in this method) is assigned to each data to be classified. Then, the image classification unit 13 calculates the center of gravity of each class for data scattered in the image feature space, and repeatedly changes the classification of each data to a cluster with a class center of gravity that is closest to it. This allows the image annotation process to be performed quickly and lightly. Note that the image classification unit 13 may have the field worker 24 check the automatically selected abnormal images and the automatically selected normal images, and have the field worker 24 determine (correct) the labels.

[0022] The classification result storage unit 14 stores the classification results (automatic abnormal images and automatic normal images) of the image classification unit 13 in a storage unit inside or outside the learning device 1. The model learning unit 15 uses machine learning with learning data as input to create an anomaly detection model 16 that takes an image as input and outputs the state that appears in the image. For example, the model learning unit 15 generates the anomaly detection model 16 by using the classification results (automatic abnormal images and automatic normal images) recorded in the classification result storage unit 14 as input data to a supervised machine learning algorithm such as LightGBM. The anomaly detection model 16 is an image recognition model that outputs two types of states, a normal state and an abnormal state, for an input image.

[0023] The monitoring device 2 obtains the state shown in the judgment image by inputting the judgment image taken by the camera at the wastewater treatment plant into the abnormality detection model 16, and if the state is abnormal, notifies the on-site worker 24 at the wastewater treatment plant by displaying or sounding a warning. For this purpose, the monitoring device 2 includes an image acquisition unit 21, an image recognition unit 22, and an abnormality notification unit 23. The image acquisition unit 21 acquires a captured image from the monitoring camera 20. The image recognition unit 22 inputs the captured image from the image acquisition unit 21 into the abnormality detection model 16, thereby recognizing the state of the drainage water (normal state or abnormal state) shown in the captured image.

[0024] When an abnormal condition is recognized by the image recognition unit 22, the abnormality notification device 23 notifies the site worker 24 of that fact and detailed information about the abnormal condition (such as the location, time period, and reason) output by the image recognition unit 22. The abnormality notification device 23 may be, for example, a device that notifies of an emergency condition by sound or light, such as a patrol lamp or buzzer, or may be a device that provides a high level of information at a glance, such as a monitoring monitor.

[0025] Furthermore, the monitoring monitor can inform the field worker 24 of specific future measures by displaying a warning screen that displays a captured image that the image recognition unit 22 has recognized as an abnormal state alongside a countermeasure to improve the abnormal state. Therefore, in addition to the abnormal state, the manual abnormality image is also annotated with a method for dealing with the abnormal state. When labeling an unknown image similar to the manual abnormal image with an abnormal state, the image classification unit 13 also labels the unknown image with a method for dealing with the abnormal state. The model learning unit 15 creates an abnormality detection model 16 that associates not only the abnormal state but also the method for dealing with the abnormal state. Furthermore, the monitoring device 2 displays a warning screen that displays the image for determination, the abnormal state shown in the image for determination, and how to deal with the abnormal state side by side.

[0026] FIG. 4 is a flowchart showing an outline of the processing performed by the learning device 1. The image acquisition unit 11 receives an input of an acquired image from the learning camera 10 (S101). The image preprocessing unit 12 classifies the images acquired in S101 into unknown images and manually abnormal images based on annotations (labels) input by the field worker 24 (S102). The image preprocessing unit 12 extracts image features from each of the images (the unknown image and the manual abnormal image) in S102 (S103).

[0027] The image feature amounts extracted in S103 are exemplified below. [Features Indicating the Color of the Wastewater] Statistical values ​​of brightness (maximum, minimum, average, variance, etc.) are extracted as image features from each channel of each image (in the case of an RGB color model, a total of three channels: R channel, G channel, and B channel). In other words, the image preprocessing unit 12 extracts statistical values ​​of brightness for each color component contained in the image as image features. This allows the state of the wastewater's component value (pH value) to be determined from the color of the wastewater indicated by the brightness statistics, so that, for example, for wastewater with a high pH value, the abnormality notification device 23 can present to the on-site worker 24 countermeasures such as increasing the amount of pH adjuster added.

[0028] [Features indicating the size of sludge contained in wastewater] Information on particles indicating sludge (particle size, particle flow speed, etc.) is extracted from each image as image features. This allows the system to detect abnormal conditions such as slow drainage flow causing stagnation, and the abnormality notification device 23 can then suggest to the on-site worker 24 how to deal with drainage with a high pH value, for example, by increasing the pump's flow rate to expedite drainage.

[0029] [Features indicating the flow of flowing water] When each image contains continuous time-series changes, such as in video recording, statistical values ​​of brightness (maximum, minimum, average, variance, etc.) are extracted as image features from the difference data between images. This reduces the amount of calculation required for the classification process (S104) in the image classification unit 13.

[0030] The image classification unit 13 classifies the feature amounts (data) of each image into provisional classes (clusters) using a clustering method such as the K-means method, and stores the classification results in the classification result storage unit 14 (S104, details in FIG. 5). The provisional classes are classified into either a "provisional normal class" in which the unknown image to which it belongs is automatically estimated to be a normal image, or a "provisional abnormal class" in which the unknown image to which it belongs is automatically estimated to be an abnormal image.

[0031] In the classification process of S104, a recursive classification process is performed such that one provisional class is classified to include a plurality of data, and then the provisional class is divided into a plurality of provisional classes. That is, the image classification unit 13 further performs a process of dividing the generated class containing the most manual abnormal images into a plurality of classes by clustering. Therefore, even with the same data, the relationship between data and classes may change, such that the data may belong to a provisional normal class at one stage, but may then belong to the provisional normal class at the next stage. For this reason, the class is named "provisional class."

[0032] In this way, the image classification unit 13 can reduce the amount of calculations and prevent a low-accuracy number of classes from being specified by performing recursive classification processing for clustering with the number of classes = 2 until the termination condition is met, without determining the number of classes in advance. Furthermore, the more classes are divided, the closer the data belonging to a provisional class is to the center of gravity of the provisional class, thereby improving the accuracy of annotation.

[0033] The image classification unit 13 automatically labels the unknown image as a normal image or an abnormal image according to the provisional class classified in S104 (S105). Note that, due to the recursive classification process, the number of times the same data belongs to the provisional normal class (normal count) and the number of times it belongs to the provisional abnormal class (abnormal count) may be stored in the classification result storage unit 14. Therefore, the image classification unit 13 automatically classifies each data (each unknown image) into either an automatic normal image or an automatic abnormal image, for example, by the following method. If the number of normal images is greater than the number of abnormal images, the image will automatically be considered normal. A list of each unknown image sorted in descending order of the number of abnormalities is displayed, and the image is automatically classified as either a normal image or an abnormal image based on the confirmation result of the field worker 24. In other words, semi-automatic annotation is performed so that the field worker 24 can correct the labels assigned by the image classification unit 13.

[0034] The model learning unit 15 outputs the anomaly detection model 16 created by machine learning using the automatic normal images and automatic abnormal images labeled in S105 as supervised learning data to the monitoring device 2 (image recognition unit 22) (S106).

[0035] Fig. 5 is a flowchart showing the details of the process (S104) of classifying the feature quantities of each image into provisional classes in Fig. 4. This flowchart starts in a state where the correspondence between the provisional classes and the images to which they belong, which has been previously recorded in the classification result storage unit 14, has been initialized. The image classification unit 13 classifies each image feature (data) extracted in S103 into a provisional class by clustering (S201). The image classification unit 13 starts a loop for sequentially selecting the provisional classes classified in S201 (S202). The image classification unit 13 calculates the percentage of manual abnormal images that belong to the selected provisional class (S203). For example, if there are 3 manual abnormal images and 7 unknown images, the percentage of manual abnormal images is 3÷(3+7)=30%.

[0036] The image classification unit 13 determines whether the percentage of manual abnormal images calculated in S203 is higher than a predetermined percentage (for example, 25%) (S204). If S204 is Yes (many), the image classification unit 13 determines that the selected provisional class is the provisional abnormal class, and records the correspondence between the provisional abnormal class and the images belonging to it in the classification result storage unit 14 (S205).If S204 is No (few), the image classification unit 13 determines that the selected provisional class is the provisional normal class, and records the correspondence between the provisional normal class and the images belonging to it in the classification result storage unit 14 (S206). This ends the loop from S202 (S207).

[0037] Here, the image classification unit 13 determines whether or not to end the recursive classification process (S211). If the classification process is not to be ended (No in S211), the image classification unit 13 selects a division target class from the provisional abnormal classes that have already been classified (S212), and returns to the process of classifying the division target class into provisional classes (S201). The division target class is selected, for example, as the provisional abnormal class to which the number or proportion of manual abnormal images belonging is the largest. That is, the image classification unit 13 recursively repeats the process of creating multiple (two) temporary child classes by dividing the temporary parent abnormal class by performing clustering again using the K-means method on the temporary parent abnormal class that has already been classified. This allows classification to be performed on a group of images from which automatic normal images have been excluded.

[0038] The termination condition of S211 is, for example, one of the following. (Condition 1) A condition for ending the process when the proportion of manually abnormal images included in the provisional normal class is greater than a predetermined proportion (for example, 10%). (Condition 2) A condition for ending the process when the number of manually abnormal images included in the provisional normal class is greater than a predetermined number (for example, three). (Condition 3) The distance between the centers of gravity of the provisional normal class and the provisional abnormal class is calculated, and if it is less than a predetermined distance, the process is terminated. The parameters of these termination conditions (predetermined ratio, predetermined number, predetermined distance) can be set by the user so as not to terminate the recursive classification process prematurely. Furthermore, when the termination condition of S211 is satisfied, the image classification unit 13 may switch the image feature extracted in S103 to another image feature and continue the recursive classification process instead of terminating the recursive classification process.

[0039] 5, each image is classified at least once into a provisional normal class or a provisional abnormal class. Therefore, the image classification unit 13 reads out the provisional class to which the image belongs (the correspondence between the class and the data) classified in S104 from the classification result storage unit 14, and automatically labels the unknown image as a normal image or an abnormal image (S105). The image classification unit 13 classifies an unknown image that has been classified into the provisional abnormal class a predetermined number of times or more into an automatic abnormal image similar to a manual abnormal image. The image classification unit 13 classifies an unknown image that has been classified into the provisional abnormal class less than a predetermined number of times into an automatic abnormal image that is similar to a manual normal image. The site worker 24 designates the predetermined number of times. Alternatively, the image classification unit 13 may display on the abnormality notification device 23 a list of unknown images sorted in descending order of the number of times they have been classified into provisional abnormal classes, and classify the unknown images selected by the field worker 24 into automatic abnormal images similar to the manual abnormal images (semi-automatic classification).

[0040] FIG. 6 is an explanatory diagram showing the process of clustering processing (S104) in the image feature space. 6, as an example of the image feature space, the horizontal axis represents the first feature (maximum luminance value of the R channel) and the vertical axis represents the second feature (average luminance value of the R channel). On the other hand, the image feature space may be configured as an arbitrary vector space with one or more image feature values ​​as axes (dimensions). The state of the image feature space transitions in the order of a spatial graph 110 before the clustering process, a spatial graph 120 after the first clustering process, and a spatial graph 130 after the second clustering process.

[0041] The spatial graph 110 is dotted with ten unknown images B1 to B10 and four manual abnormal images W1 to W4. In the spatial graph 110, a provisional normal class C11B to which the five unknown images B1 to B5 belong, and a provisional abnormal class C12W to which the five unknown images B6 to B10 and the four manual abnormal images W1 to W4 belong are formed. In the spatial graph 120, the provisional normal class C11B remains unchanged, and the provisional abnormal class C12W is divided into a provisional normal class C13B and a provisional abnormal class C14W. The provisional normal class C13B includes three unknown images B6 to B8 and one manual abnormal image W1. The provisional abnormal class C14W includes two unknown images B9 and B10 and three manual abnormal images W2 to W4.

[0042] The image classification unit 13 ends the clustering process in the state of spatial graph 120 as the termination condition of S211 (Condition 1) since the proportion (1 / 4 = 25%) of manual abnormal images included in the provisional normal class C13B is greater than a predetermined proportion (for example, 10%). Then, the image classification unit 13 classifies the unknown images (unknown images B9, B10) that have been classified into the provisional abnormal class a predetermined number of times (for example, twice) or more into automatic abnormal images similar to the manual abnormal images, and classifies the remaining unknown images (unknown images B1 to B8) into automatic normal images.

[0043] FIG. 7 is an explanatory diagram showing an example of a captured image. The automatic normal image 301 and the automatic abnormal image 302 are the results of images of the wall surface of the settling tank 204 being annotated by the image recognition unit 22. In the automatic normal image 301, a large amount of coarse-grained sludge is scattered within the area 301A, and the sludge has been properly removed from the wastewater by the settling tank 204, resulting in a normal state with little sludge being discharged to the outside of the site 212. In the automatic abnormal image 302, no sludge is found within the area 302A, resulting in an abnormal state in which the sludge contained in the wastewater flows directly into the outside of the site 212. Furthermore, the automatic normal image 301 has more black components of sludge than the automatic abnormal image 302.

[0044] FIG. 8 is a graph showing the results of analysis performed by the model learning unit 15 on the captured image of FIG. Graph 311 is a histogram extracted from the luminance components of each pixel in the automatic normal image 301. Graph 312 is a histogram extracted from the luminance components of each pixel in the automatic abnormal image 302. The range of RGB brightness values ​​"0-70" (dashed rectangle) in each graph corresponds to the black component in the image. Graph 311 of automatic normal image 301 has more black components of sludge than graph 312 of automatic abnormal image 302. Therefore, the model learning unit 15 creates an anomaly detection model 16 that determines that the captured image is in a normal state as the black components of sludge appear in the captured image increase.

[0045] FIG. 9 is a hardware configuration diagram of each device (learning device 1, monitoring device 2) of the determination system 100. Each device (learning device 1, monitoring device 2) of the determination system 100 is configured as a computer 900 having a CPU 901, RAM 902, ROM 903, HDD 904, communication I / F 905, input / output I / F 906, and media I / F 907. The communication I / F 905 is connected to an external communication device 915. The input / output I / F 906 is connected to an input / output device 916. The media I / F 907 reads and writes data from a recording medium 917. Furthermore, the CPU 901 executes a program (also called an application or an app for short) loaded into the RAM 902 to improve and control each processing unit. This program can be distributed via a communication line or recorded on a recording medium 917 such as a CD-ROM and distributed.

[0046] The determination system 100 of this embodiment described above automates the annotation work of captured images so that the learning device 1 can efficiently learn the anomaly detection model 16 that the monitoring device 2 uses to determine the state of images captured by the monitoring camera 20. To this end, the learning device 1 classifies unknown images that are similar to manual abnormal images into automatic abnormal images. Note that the similarity relationship between images can be obtained by recursively repeating the operation of classifying each image into a provisional class (provisional normal class, provisional abnormal class) using clustering (K-means method) with the manual abnormal image and the unknown image as input.

[0047] Furthermore, the learning device 1 reduces the computational load required for automating annotation by clustering image features using the lightweight K-means algorithm. Therefore, the learning device 1 can learn (or update) the anomaly detection model 16 in real time on a resource-saving device such as an embedded computer with limited computational resources that is installed near infrastructure facilities. On the other hand, Patent Document 1 does not automate annotation but uses unsupervised learning using convolutional AE. Therefore, the technology of repeatedly performing such convolution processing imposes a high computational load, making it difficult to execute in real time or on a device with limited computational resources, such as an embedded computer that operates in an environment physically close to the target of anomaly detection.

[0048] Furthermore, the present invention is not limited to the above-described embodiments, and various other applications and modifications are possible without departing from the spirit of the present invention as defined in the claims. For example, the above-described embodiments provide a detailed and specific description of the configuration of the determination system 100 in order to clearly explain the present invention, and the present invention is not necessarily limited to a system including all of the components described. Furthermore, it is possible to replace part of the configuration of one embodiment with a component of another embodiment. It is also possible to add a component of another embodiment to the configuration of one embodiment. It is also possible to add, replace, or delete other components from part of the configuration of each embodiment.

[0049] Furthermore, the above-described configurations, functions, processing units, etc. may be partially or entirely implemented in hardware, for example, by designing them as integrated circuits. As the hardware, a broad processor device such as an FPGA (Field Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit) may be used. Furthermore, each component of the determination system 100 according to the above-described embodiment may be implemented in any hardware as long as the respective hardware can transmit and receive information to and from each other via a network. Furthermore, the processing performed by a certain processing unit may be realized by a single piece of hardware, or may be realized by distributed processing using multiple pieces of hardware. [Explanation of symbols]

[0050] 1 Learning device 2 Monitoring device (monitoring section) 10 Educational Camera 11 Image acquisition unit 12 Image preprocessing section 13 Image Classification Unit 14 Classification result storage unit 15 Model Learning Section 16 Anomaly detection model 20 Surveillance Cameras 21 Image acquisition unit 22 Image Recognition Unit 23 Abnormality notification device 24 Field Workers 100 Judgment System B1~B10 Unknown images W1~W4 Manual abnormal image (confirmed abnormal image)

Claims

1. The image classification unit generates learning data for determining the state shown in an image by labeling the unknown image similar to the confirmed abnormal image as an abnormal state and labeling the unknown image not similar to the confirmed abnormal image as a normal state, based on the confirmed abnormal image labeled as showing an abnormal state and the unknown image whose state is unknown. Judging system.

2. The system further comprises a model learning unit that creates an anomaly detection model that receives an image as an input and outputs the image state by machine learning using the learning data as an input. The determination system according to claim 1 .

3. The determination system further includes an image preprocessing unit that extracts image features of the confirmed abnormal image and the unclear image, The image classification unit generates a class to which each image belongs by clustering an image feature space in which the image feature of each image extracted by the image preprocessing unit is arranged, and determines whether the unknown image is similar to the confirmed abnormal image according to the proportion of the confirmed abnormal images included in the generated class. The determination system according to claim 2 .

4. The image preprocessing unit extracts a statistical value of brightness for each color component included in the image as an image feature. The determination system according to claim 3 .

5. The image classification unit further performs a process of dividing the generated class containing the most confirmed abnormal images into a plurality of classes by the clustering. The determination system according to claim 3 .

6. The system is characterized in that it has a monitoring unit that inputs a judgment image taken by a camera at a wastewater treatment plant into the anomaly detection model, acquires the state shown in the judgment image, and outputs a warning display or a warning sound if the state is an abnormal state. The determination system according to claim 2 .

7. the anomaly detection model is capable of outputting a plurality of types of abnormal states in a manner that allows them to be distinguished from one another; The monitoring unit outputs a plurality of types of abnormal states shown in the determination image output from the anomaly detection model in a distinguishable manner. The determination system according to claim 6 .

8. The confirmed abnormal image is associated with not only an abnormal state but also a method for dealing with the abnormal state, When labeling the unknown image similar to the confirmed abnormal image with an abnormal state, the image classification unit also labels the unknown image with a method for dealing with the abnormal state, the model learning unit creates the anomaly detection model in which, in addition to an abnormal state, a method of dealing with the abnormal state is associated with the anomaly detection model; The monitoring unit displays a warning screen that displays the image for determination, an abnormal state shown in the image for determination, and a method for dealing with the abnormal state. The determination system according to claim 6 .

9. The determination system includes an image classification unit, The image classification unit creates learning data for determining the state shown in an image by labeling the unknown images similar to the confirmed abnormal images as an abnormal state and labeling the unknown images not similar to the confirmed abnormal images as a normal state, based on confirmed abnormal images labeled as showing an abnormal state and unknown images whose state is unknown. Judgment method.

Images