Abnormality detection device, abnormality detection method, and abnormality detection program

Abstract

An abnormality detection device (100) detects abnormalities at infrastructure. An inspection target detection unit (11) detects an inspection target location for abnormality detection from an inspection image. A moving object detection unit (114) detects moving objects that are included in the inspection image. An outside-inspection removal unit (116) removes the moving objects and the area outside of the inspection target location from the inspection image as outside-inspection objects and acquires an inspection area image. An abnormality detection modeling unit (123) generates a normal model that represents a normal state for the inspection target location on the basis of a plurality of inspection area images.

Description

Anomaly detection device, anomaly detection method, and anomaly detection program

[0001] The present disclosure relates to an anomaly detection device, an anomaly detection method, and an anomaly detection program, and more particularly to an anomaly detection device, an anomaly detection method, and an anomaly detection program that detect an anomaly in an infrastructure.

[0002] Recently, there has been a demand for reducing the costs of maintaining infrastructure such as bridges, roads, tunnels, and railways. To address this, attempts are being made to use vehicles or drones equipped with sensors such as cameras to photograph infrastructure and automatically detect abnormalities or areas of deterioration due to aging from the photographic data.

[0003] Patent Document 1 discloses a technique for analyzing cracks in roads using images taken by a camera attached to a vehicle.

[0004] Japanese Patent Application Laid-Open No. 2020-056303

[0005] For example, there is a technology that uses a machine learning method such as CNN to learn a damage detection model and then uses the learned model during inference to estimate abnormal locations. CNN is an abbreviation for Convolutional Neural Network. Damage can have a variety of patterns. For this reason, it is difficult to acquire such patterns using machine learning methods that use supervised learning. Therefore, the estimation accuracy of machine learning methods that use supervised learning is not necessarily high. Furthermore, when using machine learning methods that use supervised learning, a large number of images with the positions of cracks annotated are required, which poses a problem of increased manual costs.

[0006] The present disclosure aims to detect anomalies in infrastructure with high accuracy and low cost.

[0007] The anomaly detection device according to the present disclosure comprises an inspection target detection unit that acquires an inspection image capturing an area including an inspection target location that is the subject of anomaly detection, and detects the inspection target location from the inspection image; a moving object detection unit that detects moving objects included in the inspection image; an outside-inspection removal unit that removes, from the inspection image, areas other than the inspection target location and the moving objects as outside-inspection objects, and acquires an inspection range image obtained by removing the outside-inspection objects; and an anomaly detection modeling unit that acquires a plurality of the inspection range images, and generates a normal model that represents a normal state in the inspection target location based on the plurality of inspection range images.

[0008] The anomaly detection device according to the present disclosure obtains multiple inspection range images by removing areas other than the inspection target area and moving objects from the inspection image, and generates a normal model from the multiple inspection range images. The anomaly detection device according to the present disclosure performs an anomaly detection process using such a normal model, thereby achieving the effect of enabling highly accurate anomaly detection to be performed at low cost.

[0009] 1 is a diagram showing an example of the hardware configuration of an anomaly detection device according to embodiment 1. FIG. 2 is a diagram showing an example of the functional configuration of an anomaly detection device according to embodiment 1. FIG. 3 is a flow diagram showing an example of the operation of pre-processing according to embodiment 1. FIG. 4 is a flow diagram showing an example of the operation of learning processing in the anomaly information detection processing according to embodiment 1. FIG. 5 is a flow diagram showing an example of the operation of inference processing in the anomaly information detection processing according to embodiment 1. FIG. 6 is a diagram showing an example of the hardware configuration of an anomaly detection device according to a modification of embodiment 1. FIG. 7 is a diagram showing an example of the functional configuration of an anomaly detection device according to embodiment 2. FIG. 8 is a flow diagram showing an example of the operation of pre-processing according to embodiment 2. FIG. 9 is a flow diagram showing an example of the operation of learning processing in the anomaly information detection processing according to embodiment 2. FIG. 10 is a flow diagram showing an example of the operation of inference processing in the anomaly information detection processing according to embodiment 2.

[0010] The present embodiment will be described below with reference to the drawings. In each drawing, the same or corresponding parts are assigned the same reference numerals. In the description of the embodiment, the description of the same or corresponding parts will be omitted or simplified as appropriate. The arrows in the drawings mainly indicate the flow of data or the flow of processing.

[0011] Embodiment 1. ***Description of Configuration*** Fig. 1 is a diagram illustrating an example of the hardware configuration of an anomaly detection device 100 according to this embodiment. Fig. 2 is a diagram illustrating an example of the functional configuration of the anomaly detection device 100 according to this embodiment.

[0012] The anomaly detection device 100 is a computer. The anomaly detection device 100 includes a processor 910 as well as other hardware such as a memory 921, an auxiliary storage device 922, an input / output interface 930, and a communication interface 950. The processor 910 is connected to the other hardware via a signal line 80 and controls the other hardware.

[0013] 2 , the anomaly detection device 100 includes, as functional elements, a pre-processing unit 110, an anomaly information detection unit 120, and an information organizing and displaying unit 130. The pre-processing unit 110 includes an inspection target detection unit 11, a moving object detection unit 114, a front-facing image generation unit 115, an out-of-inspection removal unit 116, and a normalization unit 117. The inspection target detection unit 11 includes a white line detection unit 111, a road area detection unit 112, and a method switching unit 113. The anomaly information detection unit 120 includes an attribute classification unit 121, an anomaly detection modeling unit 123, and a threshold application unit 124. The information organizing and displaying unit 130 includes an anomaly information creation unit 131 and an anomaly information presentation unit 132.

[0014] The processor 910 is a device that executes an anomaly detection program. The anomaly detection program is a program that realizes the functions of the anomaly detection device 100. The processor 910 is an IC that performs arithmetic processing. Specific examples of the processor 910 are a CPU, a DSP, and a GPU. IC is an abbreviation for Integrated Circuit. CPU is an abbreviation for Central Processing Unit. DSP is an abbreviation for Digital Signal Processor. GPU is an abbreviation for Graphics Processing Unit.

[0015] The memory 921 is a storage device that temporarily stores data. Specific examples of the memory 921 are SRAM and DRAM. SRAM is an abbreviation for Static Random Access Memory. DRAM is an abbreviation for Dynamic Random Access Memory. The auxiliary storage device 922 is a storage device that saves data. A specific example of the auxiliary storage device 922 is an HDD. The auxiliary storage device 922 may also be a portable storage medium such as an SD (registered trademark) memory card, CF, NAND flash, flexible disk, optical disk, compact disk, Blu-ray (registered trademark) disk, or DVD. Note that HDD is an abbreviation for Hard Disk Drive. SD (registered trademark) is an abbreviation for Secure Digital. CF is an abbreviation for CompactFlash (registered trademark). DVD is an abbreviation for Digital Versatile Disk.

[0016] The input / output interface 930 includes an input interface and an output interface. The input interface is an interface for connecting an input device. The output interface is an interface for connecting an output device. Specific examples of the input / output interface 930 include USB and HDMI (registered trademark) ports. USB is an abbreviation for Universal Serial Bus. HDMI (registered trademark) is an abbreviation for High-Definition Multimedia Interface.

[0017] To the input interface, for example, a camera and a GPS are connected as input devices. GPS is an abbreviation for Global Positioning System. Inertial measurement devices such as an acceleration sensor and a gyro sensor may also be connected to the input interface as input devices. LiDAR may also be connected to the input interface as input devices. To the output interface, for example, a display and a speaker are connected as output devices.

[0018] The communication interface 950 is an interface for communicating with an external device that stores maps, etc. Specific examples of the communication interface 950 include an Ethernet (registered trademark) port or a device that performs wireless communication.

[0019] The anomaly detection program is executed in the anomaly detection device 100. The anomaly detection program is loaded into the processor 910 and executed by the processor 910. In addition to the anomaly detection program, an OS is also stored in the memory 921. OS is an abbreviation for Operating System. The processor 910 executes the anomaly detection program while executing the OS. The anomaly detection program and the OS may be stored in an auxiliary storage device 922. The anomaly detection program and the OS stored in the auxiliary storage device 922 are loaded into the memory 921 and executed by the processor 910. Note that part or all of the anomaly detection program may be incorporated into the OS.

[0020] The anomaly detection device 100 may include multiple processors that replace the processor 910. These multiple processors share the task of executing the anomaly detection program. Each processor is a device that executes the anomaly detection program in the same way as the processor 910.

[0021] The data, information, signal values ​​and variable values ​​used, processed or output by the anomaly detection program are stored in memory 921, auxiliary storage device 922, or registers or cache memory within processor 910.

[0022] The "part" of each part of the anomaly detection device 100 may be read as a "circuit," "step," "procedure," "process," or "circuitry." The anomaly detection program causes a computer to execute each process, where the "part" of each part of the anomaly detection device 100 is read as a "process." The "process" of each process of the anomaly detection device 100 may be read as a "program," "program product," "computer-readable storage medium storing a program," or "computer-readable recording medium recording a program." Furthermore, the anomaly detection method is a method performed by the anomaly detection device 100 executing the anomaly detection program. The anomaly detection program may be provided by being stored in a computer-readable recording medium. Furthermore, the anomaly detection program may be provided as a program product.

[0023] ***Description of Operation*** Next, the operation of the anomaly detection device 100 according to this embodiment will be described. The operating procedure of the anomaly detection device 100 corresponds to an anomaly detection method. Furthermore, a program that realizes the operation of the anomaly detection device 100 corresponds to an anomaly detection program. The anomaly detection process by the anomaly detection device 100 comprises pre-processing by the pre-processing unit 110, anomaly information detection process by the anomaly information detection unit 120, and information organization and display process by the information organization and display unit 130.

[0024] <Pre-processing> An overview of the pre-processing by the pre-processing unit 110 will be described. The inspection target detection unit 11 acquires an inspection image 31 obtained by capturing an area including an inspection target area 40 that is the target of anomaly detection, and detects the inspection target area 40 from the inspection image 31. In this embodiment, the inspection target detection unit 11 acquires an inspection image 31 obtained by capturing the road surface as the inspection target area 40. The inspection target detection unit 11 then detects the road as the inspection target area 40 from the inspection image 31 by switching between a white line detection method that detects white lines on the road surface from the inspection image 31 and a road area classification method that classifies the road area from the inspection image. The inspection target area 40 is also referred to as the inspection target range. The white line detection method is performed by the white line detection unit 111. The road area classification method is performed by the road area detection unit 112.

[0025] The moving object detection unit 114 detects moving objects included in the inspection image 31. The non-inspection removal unit 116 removes from the inspection image 31 the area other than the inspection target area 40 and moving objects as non-inspection objects 41, and acquires the inspection range image 32 obtained by removing the non-inspection objects 41. Specifically, this is as follows.

[0026] 3 is a flow diagram showing an example of the operation of pre-processing according to this embodiment. In step S101, the white line detection unit 111 of the inspection object detection unit 11 performs a white line detection method to detect white lines on the road surface for the inspection image 31 obtained by capturing the road surface. As an example, the white line detection unit 111 performs a white line detection method using deep learning such as CNN. In step S102, the method switching unit 113 of the inspection object detection unit 11 determines whether or not a white line has been detected by the white line detection method. If a white line has been detected, the method switching unit 113 proceeds to step S104. If a white line has not been detected, the method switching unit 113 proceeds to step S103.

[0027] In step S103, the road area detection unit 112 performs a road area classification method on the inspection image 31 to classify road areas. For example, the road area detection unit 112 identifies gray road areas. As an example, the road area detection unit 112 performs the road area classification method using a method that uses deep learning, such as CNN. As described above, when the white line detection unit 111 cannot detect white lines, the method switching unit 113 switches the method, for example, detecting roads using the road area classification method used by the road area detection unit 112.

[0028] In step S104, the moving object detection unit 114 detects a moving object included in the inspection image 31. The moving object is an object that is not subject to learning and estimation performed in the anomaly information detection process described below. The moving object is, for example, a passing vehicle such as a truck, a regular vehicle, a bicycle, or a motorcycle. In step S105, the front-facing image generation unit 115 converts the inspection image 31 into a front-facing image viewed from a position directly facing the inspection target area 40. Specifically, the front-facing image generation unit 115 generates a front-facing image from the inspection image 31 using the road or white lines detected in steps S101 to S103. The front-facing image is, for example, an image such as a bird's-eye view image or a top-view image.

[0029] In step S106, the non-inspection removal unit 116 removes areas other than the inspection target location 40 and moving objects from the inspection image 31 converted into a front-facing image as non-inspection objects 41. Specifically, the non-inspection removal unit 116 sets areas outside the road or outside the white lines and moving objects as non-inspection objects 41. The non-inspection removal unit 116 removes the non-inspection objects 41 from the inspection image 31 converted into a front-facing image, and obtains the inspection range image 32. In step S107, the normalization unit 117 normalizes the inspection range image 32 to convert it into an image with standard brightness.

[0030] <Abnormality Information Detection Processing> Next, an overview of the abnormality information detection processing performed by the abnormality information detection unit 120 will be described. The abnormality detection modeling unit 123 of the abnormality information detection unit 120 acquires a plurality of inspection range images 32 and generates a normal model 50 for the inspection target location 40 based on the plurality of inspection range images 32. The normal model 50 represents a normal state of the inspection target location 40. The abnormality detection modeling unit 123 also calculates the distance between the normal model 50 and the inspection range image 32 and performs processing to detect an abnormality in the inspection range image 32 based on the distance. In this way, the abnormality information detection unit 120 performs a learning processing to learn the normal model 50 using the inspection range image 32 obtained by the pre-processing unit 110, and an inference processing to infer and detect an abnormality using the normal model 50. Specifically, the processing is as follows.

[0031] <<Learning Process>> FIG. 4 is a flow diagram showing an example of the operation of the learning process of the anomaly information detection process according to this embodiment. In step S201, the attribute classification unit 121 acquires multiple inspection range images 32. The attribute classification unit 121 classifies the multiple inspection range images 32 based on the attributes of the inspection target locations 40. For example, the attribute classification unit 121 classifies the inspection range images 32 based on attributes such as road position information. Alternatively, the attribute classification unit 121 may classify the inspection range images 32 based on attributes such as road identifiers that identify roads. Specifically, the attribute classification unit 121 classifies the multiple inspection range images 32 by performing unsupervised learning on the multiple inspection range images 32 in units of roads or regions obtained based on position information. Examples of unsupervised learning include K-means, DBSCAN, and primary component analysis. In this way, the attribute classification unit 121 classifies the multiple inspection range images 32 on an image-by-image basis. This classification is used to switch the normal model 50 used for anomaly detection based on the road identifier or position information. Hereinafter, this classification will be referred to as road attribute classification.

[0032] The processes from step S203 to step S206 are repeatedly executed the number of times equal to the number of classes in the road attribute classification (step S202).

[0033] In step S203, the partial image classification unit 122 classifies the inspection range image 32 into multiple partial images based on the characteristics of the partial images included in the inspection range image 32. For example, the partial image classification unit 122 classifies the inspection range image 32 into multiple partial images based on the characteristics of the road included in the inspection range image 32. The road characteristics are properties such as the road pattern or road color. Specifically, the inspection range image 32 includes multiple partial images divided into a grid. The partial image classification unit 122 classifies the multiple partial images based on the road characteristics. Many anomaly detection algorithms perform anomaly detection for each partial image obtained by dividing an image into regions. For example, anomaly detection or threshold learning is performed according to the pattern of the partial image. If the pattern of a partial image differs from that of a normal image, it may be excluded from anomaly detection. More specifically, the partial image classification unit 122 classifies the multiple partial images divided into grids by performing unsupervised learning on a partial image basis. Examples of unsupervised learning include K-means, DBSCAN, Primary Component Analysis, etc. Classification of partial images is called partial image classification.

[0034] The processes from step S205 to step S206 are repeatedly executed the number of times equal to the number of classes in the partial image classification (step S204).

[0035] In step S205, the anomaly detection modeling unit 123 generates a normal model for the corresponding partial image by learning using multiple inspection range images 32 that belong to the same road attribute classification. Here, any anomaly detection method can be used to generate the normal model. The anomaly detection modeling unit 123 creates the normal model using an anomaly detection method such as PaDiM or PatchCore, for example.

[0036] In step S206, the threshold application unit 124 sets a threshold for each of the multiple partial images by learning using multiple inspection range images 32 that belong to the same road attribute classification. The threshold is a distance threshold used in the inference process. When the processes from step S205 to step S206 are completed for all classes of the partial image classification, the same processes are repeated using the inspection range images 32 in the next class of the road attribute classification.

[0037] As described above, through the learning process from step S201 to step S206, the anomaly detection modeling unit 123 uses multiple inspection range images 32 to generate a normal model 50 for each attribute of the inspection target area 40 and for each characteristic of the partial image.

[0038] 5 is a flow diagram showing an example of the operation of the inference process of the anomaly information detection process according to this embodiment. In step S301, the anomaly detection modeling unit 123 acquires one inspection range image 32. The attribute classification unit 121 identifies the attribute of the inspection range image 32 based on the attribute of the inspection target portion 40 in the inspection range image 32, and classifies the inspection range image 32. The method of classifying the inspection range image 32 is the same as in step S201.

[0039] In step S302, the anomaly detection modeling unit 123 acquires a normal model corresponding to the attributes of the inspection target area 40. Here, the anomaly detection modeling unit 123 acquires a normal model for each partial image, i.e., a plurality of normal models corresponding to the attributes of the inspection target area 40. The anomaly detection modeling unit 123 calculates the distance between the inspection range image 32 and the normal model corresponding to that attribute. Specifically, the anomaly detection modeling unit 123 calculates the distance between each partial image of the inspection range image 32 and the normal model corresponding to that partial image. In other words, the anomaly detection modeling unit 123 calculates the distance by replacing the normal model for each partial image of the inspection range image 32.

[0040] In step S303, the anomaly detection modeling unit 123 detects an anomaly in the inspection range image 32 by comparing the distance for each partial image between the inspection range image 32 and the normal model with a threshold value for each partial image. In this embodiment, it is assumed that there are multiple partial images of the normal model, and a threshold value is set for each partial image. The anomaly detection modeling unit 123 detects an anomaly in a partial image in the inspection range image 32 using the distance for each partial image and the threshold value for each partial image.

[0041] <Information Organizing and Display Processing> Next, the information organizing and display processing by the information organizing and display unit 130 will be described. The information organizing and display unit 130 organizes and presents anomaly detection results. For example, the information organizing and display unit 130 may combine the results with location information to plot anomaly locations on a map. The information organizing and display unit 130 may also calculate an anomaly rate for each road. The information organizing and display unit 130 also organizes and presents information, such as promptly reporting major anomalies and creating a ledger.

[0042] The anomaly information creation unit 131 creates information that compiles the anomaly detection results and other information. For example, the anomaly information creation unit 131 creates the following information: - Create information that maps anomalies on a map: Anomaly information is combined with GPS location information or a map to map the anomaly locations on a map. - Create information that summarizes the anomaly rate for each location or road: The anomaly rate around a specific location is displayed based on GPS location information. Alternatively, the anomaly rate is displayed for each road using road (section) information identified from a map.

[0043] The anomaly information presenting unit 132 presents the anomaly information created by the anomaly information creating unit 131 to the user. For example, the following use cases are possible: Display on a display: The anomaly information created by the anomaly information creating unit 131 is displayed on a display. Notification using a speaker: The anomaly information created by the anomaly information creating unit 131 is notified to the user using a speaker.

[0044] ***Explanation of the Effects of This Embodiment*** In supervised learning, damaged areas are directly detected. Since damage is significantly different from the pattern on the car, damage can be detected without vehicle detection. However, supervised learning requires learning from a huge amount of training data, which increases costs. On the other hand, in this embodiment, anomaly detection is performed using a normal model, which reduces costs. In addition, moving objects can be removed from the detected image used to create the normal model, making it possible to obtain a highly accurate normal model.

[0045] Furthermore, in this embodiment, when detecting roads from an inspection image, it is possible to switch between the white line detection method and the road area classification method. Therefore, according to this embodiment, roads can be detected with high accuracy, and areas other than roads can be removed from the detection image used to create a normal model, so a highly accurate normal model can be obtained.

[0046] In addition, in this embodiment, a normal model can be created according to the attributes of the road, which is the area to be inspected. Furthermore, in this embodiment, a normal model and a threshold can be set for each characteristic of a partial image included in an inspection range image obtained by removing non-inspection objects from an inspection image. For example, preparing a single model and threshold corresponding to all roads may result in a decrease in performance. This is because roads can have a variety of patterns, such as gray-only roads, cobblestone roads, tiled roads, and mountain paths. In this embodiment, a normal model and a threshold can be generated according to the road identifier or location information, thereby enabling more accurate detection of abnormalities. The pattern may differ for each region of an image. Even in such cases, the normal model and threshold can be changed for each pattern in a partial region, which has the effect of further improving performance.

[0047] ***Other Configurations*** <Variation 1> In this embodiment, the anomaly information detection unit classifies the inspection range image based on attributes of the inspection target location, such as location information or road identifier. Furthermore, the anomaly information detection unit classifies partial images of the inspection range image based on characteristics of the partial images, such as the pattern in the partial images of the inspection range image. Then, the anomaly information detection unit generates a normal model for each attribute of the inspection target location and for each partial image of the inspection range image, and sets a threshold for each partial image of the inspection range image.

[0048] In Modification 1 of this embodiment, classification based on attributes and classification based on the characteristics of partial images may not be performed on the inspection range image. The anomaly information detection unit may generate a normal model using multiple inspection range images that have not been classified based on the attributes of the inspection target area and the characteristics of the partial image. The anomaly information detection unit may then calculate the distance between one inspection range image and the normal model and compare it with a predetermined threshold to detect an abnormality in the inspection range image. In other words, there may be only one normal model and one threshold.

[0049] <Variation 2> Constructing a normal model for each partial image, as in this embodiment, results in a large capacity. Furthermore, the overhead of switching becomes large. Therefore, the normal model may be constructed in units of the attributes of the inspection target area, i.e., road attributes. The threshold value may be set in units of the characteristics of the partial image. The anomaly information detection unit then calculates the distance between one inspection range image and the normal model corresponding to the attribute of the inspection target area. The anomaly information detection unit detects an abnormality in the inspection range image by comparing this distance with the threshold value for each partial image. In other words, there is one normal model for each attribute of the inspection target area, and multiple threshold values. In the case of Variation 2, step S205 in FIG. 4 is executed outside the loop of step S203.

[0050] <Modification 3> In the present embodiment, the inspection image is an image obtained by photographing the road surface as the inspection target location. However, the inspection image may be an image obtained by photographing a railroad rail as the inspection target location other than the road surface.

[0051] The inspection object detection unit acquires, as an inspection image, an image captured of a railway as the inspection target location. The inspection object detection unit detects the railway as the inspection target location from the inspection image by switching between a rail detection method that detects railway rails from the inspection image and a railway area classification method that classifies railway areas from the inspection image. The attribute classification unit classifies the inspection range image based on railway position information. The anomaly detection modeling unit uses multiple inspection range images to generate a normal model for each railway position information. In addition, the partial image classification unit classifies the inspection range image into multiple partial images based on the characteristics of the railway contained in the inspection range image. Other configurations and functions are the same as those of embodiment 1.

[0052] <Modification 4> In this embodiment, the functions of each unit of the anomaly detection device 100 are realized by software. As a modification, the functions of each unit of the anomaly detection device 100 may be realized by hardware. Specifically, the anomaly detection device 100 includes an electronic circuit 909 instead of the processor 910.

[0053] 6 is a diagram showing an example of the hardware configuration of an anomaly detection device 100 according to a modified example of this embodiment. The electronic circuit 909 is a dedicated electronic circuit that realizes the functions of each unit of the anomaly detection device 100. Specifically, the electronic circuit 909 is a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, a logic IC, a GA, an ASIC, or an FPGA. GA is an abbreviation for Gate Array. ASIC is an abbreviation for Application Specific Integrated Circuit. FPGA is an abbreviation for Field-Programmable Gate Array.

[0054] The functions of each part of the anomaly detection device 100 may be realized by a single electronic circuit, or may be realized by distributing the functions across multiple electronic circuits.

[0055] As another modification, some of the functions of each unit of the anomaly detection device 100 may be realized by electronic circuits, and the remaining functions may be realized by software. Also, some or all of the functions of each unit of the anomaly detection device 100 may be realized by firmware.

[0056] Each of the processor and the electronic circuit is also called a processing circuitry. That is, the functions of each part of the anomaly detection device 100 are realized by the processing circuitry.

[0057] Embodiment 2 In this embodiment, differences from and additions to embodiment 1 will be mainly described. In this embodiment, components having the same functions as those in embodiment 1 will be assigned the same reference numerals, and descriptions thereof will be omitted.

[0058] ***Description of Configuration*** FIG. 7 is a diagram showing an example of the configuration of an anomaly detection device 100 according to the present embodiment. In this embodiment, the configuration and function of the inspection target detection unit 11 differ from those in the first embodiment. The inspection target detection unit 11 acquires, as the inspection image 31, an image of an installed object around the travel path as the inspection target location 40. The inspection target detection unit 11 then detects a part of the installed object as the inspection target location 40 from the inspection image 31. Specifically, in this embodiment, the inspection target detection unit 11 acquires the inspection image 31 of an installed object around the travel path, such as a bridge or a guardrail, as the inspection target location 40. The inspection target detection unit 11 then detects a part of the installed object, such as a screw or a rope, as the inspection target location 40 from the inspection image 31.

[0059] The other configurations are basically the same as those in the first embodiment, but because the inspection target location 40 is an object installed around the travel path, the following functions differ: The attribute classification unit 121 classifies the inspection range image based on the position information of the object. The anomaly detection modeling unit 123 uses multiple inspection range images to generate a normal model for each piece of position information of the object. The partial image classification unit 122 classifies the inspection range image into multiple partial images based on the characteristics of the object included in the inspection range image. The other configurations are the same as those in the first embodiment.

[0060] ***Description of Operation*** <Pre-processing> FIG. 8 is a flow diagram showing an example of the operation of pre-processing according to this embodiment. In step S101a, the inspection target detection unit 11 acquires, as the inspection image 31, an image of an installed object around the travel path as the inspection target location 40. Then, the inspection target detection unit 11 detects a part of the installed object as the inspection target location 40 from the inspection image 31. Specifically, the inspection target detection unit 11 acquires the inspection image 31 of an installed object around the travel path, such as a bridge or a guardrail, as the inspection target location 40. Then, the inspection target detection unit 11 detects a part of the installed object, such as a screw or a rope, as the inspection target location 40 from the inspection image 31. For example, the inspection target detection unit 11 detects a part such as a screw using a deep learning method such as CNN. The inspection target detection unit 11 detects the part in the form of a bounding box, for example. In addition to the bounding box, the part may be detected using semantic segmentation.

[0061] Steps S104 to S107 are basically the same as those in embodiment 1. However, the inspection target portion 40 in the inspection image 31 is a part of an installed object such as a bridge or a guardrail.

[0062] In step S104, the moving object detection unit 114 detects a moving object included in the inspection image 31. A moving object is an object that is not subject to learning and estimation performed in the anomaly information detection process described below. A moving object is, for example, a passing vehicle such as a truck, a regular vehicle, a bicycle, or a motorcycle. In step S105, the front-facing image generation unit 115 converts the inspection image 31 into a front-facing image viewed from a position directly facing the inspection target area 40. Specifically, the front-facing image generation unit 115 generates a front-facing image of the image of the component detected in step S101a. The front-facing image generation unit 115 converts the image of the component into a front-facing image using a method such as feature point detection. The front-facing image generation unit 115 performs a homography transformation based on how the pattern of the component is reflected to create a front-facing image.

[0063] In step S106, the non-inspection removal unit 116 removes, from the inspection image 31 converted into a front-facing image, areas other than the inspection target location 40 and moving objects as non-inspection objects 41. Specifically, the non-inspection removal unit 116 removes areas other than the bounding box and moving objects as non-inspection objects 41. Alternatively, the non-inspection removal unit 116 removes, as non-inspection objects 41, areas outside the bounding box and moving objects detected by semantic segmentation. The non-inspection removal unit 116 removes the non-inspection objects 41 from the inspection image 31 converted into a front-facing image, and obtains an inspection range image 32. In step S107, the normalization unit 117 normalizes the inspection range image 32 to convert it into an image with standard brightness.

[0064] <Abnormality Information Detection Processing> The abnormality information detection processing is also basically the same as that in embodiment 1. The abnormality information detection unit 120 performs a learning processing for learning a normal model 50 using the inspection range image 32 obtained by the pre-processing unit 110, and an inference processing for inferring and detecting an abnormality using the normal model 50. However, the inspection target location 40 in the inspection image 31 is a part of the installed object. Specifically, it is as follows.

[0065] <<Learning Process>> FIG. 9 is a flow diagram showing an example of the operation of the learning process of the anomaly information detection process according to this embodiment. The process in FIG. 9 is the same as that of the first embodiment, except that the inspection target portion 40 in the inspection image 31 is a component of an installed object. In step S201a, the attribute classification unit 121 acquires multiple inspection range images 32. The attribute classification unit 121 classifies the multiple inspection range images 32 based on the attributes of the inspection target portion 40. For example, the attribute classification unit 121 classifies the inspection range images 32 based on attributes such as location information of the installed object. Alternatively, the attribute classification unit 121 may classify the inspection range images 32 based on attributes such as an installed object identifier that identifies the installed object. Specifically, the attribute classification unit 121 classifies the multiple inspection range images 32 by performing unsupervised learning on the multiple inspection range images 32 in units of location information obtained based on the location information of the installed object or in units of installed object. Examples of unsupervised learning include K-means, DBSCAN, and Primary Component Analysis. In this way, the attribute classification unit 121 classifies the multiple inspection range images 32 on an image-by-image basis. This classification is used to switch the normal model 50 used for anomaly detection based on the installation identifier or position information. Hereinafter, this classification will be referred to as installation attribute classification.

[0066] The processes from step S203 to step S206 are repeatedly executed the number of times equal to the number of classes in the installation attribute classification (step S202a).

[0067] In step S203, the partial image classification unit 122 classifies the inspection range image 32 into multiple partial images based on the characteristics of the partial images included in the inspection range image 32. For example, the partial image classification unit 122 classifies the inspection range image 32 into multiple partial images based on the image pattern, which is a characteristic included in the inspection range image 32. The characteristic is a property such as the pattern or color of the image. Specifically, the inspection range image 32 includes multiple partial images divided into a grid. The partial image classification unit 122 classifies the multiple partial images based on the image characteristics. Many anomaly detection algorithms perform anomaly detection for each partial image obtained by dividing an image into regions. For example, anomaly detection or threshold learning is performed according to the pattern of the partial image. If the pattern of a partial image differs from that of a normal image, it may be excluded from anomaly detection. More specifically, the partial image classification unit 122 classifies the multiple partial images divided into a grid by performing unsupervised learning on a partial image basis. Examples of unsupervised learning include K-means, DBSCAN, Primary Component Analysis, etc. Classification of partial images is called partial image classification.

[0068] The processes from step S205 to step S206 are repeatedly executed the number of times equal to the number of classes in the partial image classification (step S204).

[0069] In step S205, the anomaly detection modeling unit 123 generates a normal model for the corresponding partial image by learning using multiple inspection range images 32 that belong to the same installation object attribute classification. Here, any anomaly detection method may be used to generate the normal model. The anomaly detection modeling unit 123 creates the normal model using an anomaly detection method such as PaDiM or PatchCore, for example.

[0070] In step S206, the threshold application unit 124 sets a threshold for each of the plurality of partial images by learning using a plurality of inspection range images 32 belonging to the same installation attribute classification. The threshold is a distance threshold used in the inference process. When the processes from step S205 to step S206 are completed for all classes of the partial image classification, the same processes are repeated using the inspection range images 32 of the next installation attribute classification class.

[0071] As described above, through the learning process from step S201a to step S206, the anomaly detection modeling unit 123 uses multiple inspection range images 32 to generate a normal model 50 for each attribute of the inspection target area 40 and for each characteristic of the partial image.

[0072] <<Inference Processing>> Fig. 10 is a flow diagram showing an example of the operation of the inference processing of the anomaly information detection processing according to this embodiment. The processing in Fig. 10 is the same as that of embodiment 1, except that the inspection target portion 40 in the inspection image 31 is a component of the installed object. In step S301a, the anomaly detection modeling unit 123 acquires one inspection range image 32. The attribute classification unit 121 identifies the attribute of the inspection range image 32 based on the attribute of the inspection target portion 40 in the inspection range image 32, and classifies the inspection range image 32. The method of classifying the inspection range image 32 is the same as that of step S201a.

[0073] In step S302, the anomaly detection modeling unit 123 acquires a normal model corresponding to the attributes of the inspection target area 40. Here, the anomaly detection modeling unit 123 acquires a normal model for each partial image, i.e., a plurality of normal models corresponding to the attributes of the inspection target area 40. The anomaly detection modeling unit 123 calculates the distance between the inspection range image 32 and the normal model corresponding to that attribute. Specifically, the anomaly detection modeling unit 123 calculates the distance between each partial image of the inspection range image 32 and the normal model corresponding to that partial image. In other words, the anomaly detection modeling unit 123 calculates the distance by replacing the normal model for each partial image of the inspection range image 32.

[0074] In step S303, the anomaly detection modeling unit 123 detects an anomaly in the inspection range image 32 by comparing the distance for each partial image between the inspection range image 32 and the normal model with a threshold value for each partial image. In this embodiment, it is assumed that there are multiple partial images of the normal model, and a threshold value is set for each partial image. The anomaly detection modeling unit 123 detects an anomaly in a partial image in the inspection range image 32 using the distance for each partial image and the threshold value for each partial image.

[0075] <Information Organizing and Display Processing> Next, the information organizing and display processing by the information organizing and display unit 130 will be described. The information organizing and display unit 130 organizes and presents anomaly detection results. For example, the information organizing and display unit 130 may combine the results with location information to plot anomaly locations on a map. The information organizing and display unit 130 may also calculate an anomaly rate for each installed object. The information organizing and display unit 130 also organizes and presents information, such as promptly reporting major anomalies and creating a ledger.

[0076] The anomaly information creation unit 131 creates information that compiles the anomaly detection results and other information. For example, the anomaly information creation unit 131 creates the following information: - Create information that maps anomalies on a map: Anomaly information is combined with GPS location information or a map to map the anomaly locations on a map. - Create information that summarizes the anomaly rate for each location or installation: The anomaly rate around a specific location is displayed based on GPS location information. Alternatively, the anomaly rate is displayed for each installation using road (section) information identified from a map.

[0077] The anomaly information presenting unit 132 presents the anomaly information created by the anomaly information creating unit 131 to the user. For example, the following use cases are possible: Display on a display: The anomaly information created by the anomaly information creating unit 131 is displayed on a display. Notification using a speaker: The anomaly information created by the anomaly information creating unit 131 is notified to the user using a speaker.

[0078] ***Explanation of the Effects of This Embodiment*** In supervised learning, damaged areas are directly detected. Since damage is significantly different from the pattern on the car, damage can be detected without vehicle detection. However, supervised learning requires learning from a huge amount of training data, which increases costs. On the other hand, in this embodiment, anomaly detection is performed using a normal model, which reduces costs. In addition, moving objects can be removed from the detected image used to create the normal model, making it possible to obtain a highly accurate normal model.

[0079] Furthermore, in this embodiment, a normal model can be created according to the attributes of the installation, which is the area to be inspected. Furthermore, in this embodiment, a normal model and a threshold can be set for each characteristic of a partial image included in an inspection range image obtained by removing non-inspection objects from the inspection image. For example, if one model and threshold were prepared for all roads or installations, performance may be reduced. This is because conditions specific to the location or section where the road was built may exist. In this embodiment, a normal model and a threshold can be generated according to the identifier or location information of the installation, thereby enabling more accurate detection of abnormalities. The pattern may differ for each region of the image. Even in such cases, the normal model and threshold can be changed for each pattern in the partial region, which has the effect of further improving performance.

[0080] In this embodiment, similarly to the first embodiment, the first to fourth modifications can be applied.

[0081] In the first and second embodiments, it is assumed that the location information of roads or installed objects is acquired by GPS. Alternatively, location information may be acquired by other methods, such as LiDAR / Visual SLAM, or using the gyro or acceleration of an IMU. SLAM is an abbreviation for Simultaneous Localization and Mapping. IMU is an abbreviation for Inertial Measurement Unit.

[0082] In addition, in the first and second embodiments, anomalies may be detected based on shape and pattern using both LiDAR and a camera, rather than just a camera. Alternatively, shape anomalies may be measured using LiDAR alone, and the results may be presented to the user.

[0083] In addition, in the first and second embodiments, the detection results may be identified using time-series data. The detection results may be identified with the same location in the past, and the time-series changes or the extent of the damage may be displayed. Furthermore, if a location has been marked by a user as a false positive in the past, even if it has been detected, it may be determined to be a false positive if there is no change in the pattern, and the false positive may not be displayed to the user thereafter.

[0084] Furthermore, in the first and second embodiments, a filter that emphasizes damage may be introduced. For example, a filter that emphasizes damage, such as an Edge Preserving Filter, may be introduced.

[0085] In the first and second embodiments described above, each unit of the anomaly detection device has been described as an independent functional block. However, the configuration of the anomaly detection device does not have to be the same as that of the above-described embodiments. The functional blocks of the anomaly detection device may have any configuration as long as they can realize the functions described in the above-described embodiments. Furthermore, the anomaly detection device may not be a single device, but may be a system composed of multiple devices. Furthermore, multiple parts of the first and second embodiments may be combined and implemented. Alternatively, only one part of the first and second embodiments may be implemented. In addition, the first and second embodiments may be combined in any way, either as a whole or in part. That is, in the first and second embodiments, the respective embodiments may be freely combined, or any component of each embodiment may be modified, or any component of each embodiment may be omitted.

[0086] The above-described embodiment is essentially a preferred example and is not intended to limit the scope of the present disclosure, the scope of application of the present disclosure, or the scope of use of the present disclosure. The above-described embodiment can be modified in various ways as needed. For example, the procedures described using flow charts or sequence diagrams may be modified as appropriate. For example, the order of some processes may be changed, such as by reversing the processing of the front-facing image and the white line detection.

[0087] Various aspects of the present disclosure are summarized below as appendices.

[0088] (Supplementary Note 1) An anomaly detection device comprising: an inspection target detection unit that acquires an inspection image capturing an area including an inspection target location that is the target of anomaly detection, and detects the inspection target location from the inspection image; a moving object detection unit that detects moving objects included in the inspection image; an outside-inspection removal unit that removes from the inspection image an area other than the inspection target location and the moving objects as outside-inspection objects, and acquires an inspection range image obtained by removing the outside-inspection objects; and an anomaly detection modeling unit that acquires a plurality of the inspection range images and generates a normal model representing a normal state of the inspection target location based on the plurality of inspection range images. (Supplementary Note 2) The anomaly detection device according to Supplementary Note 1, wherein the anomaly detection modeling unit calculates a distance between the normal model and the inspection range image, and detects an anomaly in the inspection range image based on the distance. (Supplementary Note 3) The anomaly detection device according to Supplementary Note 2, wherein the anomaly detection modeling unit calculates a distance between the normal model and the inspection range image, and detects an anomaly in the inspection range image based on the distance. (Supplementary Note 4) The anomaly detection device comprises: a partial image classification unit that classifies the inspection range image into a plurality of partial images based on characteristics of the partial images included in the inspection range image; and a threshold application unit that sets a threshold for the distance for each of the plurality of partial images, wherein the anomaly detection modeling unit calculates a distance between a partial image in a normal model that corresponds to an attribute of the inspection target portion and a partial image in the inspection range image, and detects an abnormality in the partial image in the inspection range image using the distance and the threshold corresponding to the partial image. (Supplementary Note 5) The anomaly detection device according to Supplementary Note 4, wherein the inspection target detection unit acquires, as the inspection image, an image of a road surface taken as the inspection target portion, and detects the road from the inspection image as the inspection target portion by switching between a white line detection method that detects white lines on the road surface from the inspection image and a road area classification method that classifies the area of ​​the road from the inspection image.(Supplementary Note 6) The anomaly detection device according to Supplementary Note 5, wherein the attribute classification unit classifies the inspection range image based on position information of the road, and the anomaly detection modeling unit uses the plurality of inspection range images to generate the normal model for each piece of position information of the road. (Supplementary Note 7) The anomaly detection device according to Supplementary Note 5 or Supplementary Note 6, wherein the partial image classification unit classifies the inspection range image into a plurality of partial images based on attributes of the road included in the inspection range image. (Supplementary Note 8) The anomaly detection device according to Supplementary Note 4, wherein the inspection target detection unit acquires, as the inspection image, an image taken of a railway as the inspection target location, and detects the railway from the inspection image as the inspection target location by switching between a rail detection method that detects railway rails from the inspection image and a railway area classification method that classifies the railway area from the inspection image. (Supplementary Note 9) The anomaly detection device according to Supplementary Note 8, wherein the attribute classification unit classifies the inspection range image based on position information of the railway, and the anomaly detection modeling unit uses the plurality of inspection range images to generate the normal model for each piece of position information of the railway. (Supplementary Note 10) The anomaly detection device according to Supplementary Note 8 or Supplementary Note 9, wherein the partial image classification unit classifies the inspection range image into a plurality of partial images based on characteristics of the railway included in the inspection range image. (Supplementary Note 11) The anomaly detection device according to Supplementary Note 4, wherein the inspection target detection unit acquires, as the inspection image, an image of an installed object around a travel path as the inspection target location, and detects a part of the installed object from the inspection image as the inspection target location. (Supplementary Note 12) The anomaly detection device according to Supplementary Note 11, wherein the attribute classification unit classifies the inspection range image based on position information of the installed object, and the anomaly detection modeling unit generates the normal model for each piece of position information of the installed object using the plurality of inspection range images. (Supplementary Note 13) The anomaly detection device according to Supplementary Note 11 or Supplementary Note 12, wherein the partial image classification unit classifies the inspection range image into a plurality of partial images based on characteristics of the installed object included in the inspection range image. (Supplementary Note 14) The anomaly detection device according to any one of Supplementary Note 1 to Supplementary Note 13, further comprising a front-facing image generation unit that converts the inspection image into a front-facing image viewed from a position directly facing the inspection target location.(Supplementary Note 15) An anomaly detection method in which a computer acquires an inspection image capturing an area including an inspection target location that is the subject of anomaly detection, detects the inspection target location from the inspection image, detects a moving object included in the inspection image, removes from the inspection image an area other than the inspection target location and the moving object as non-inspection objects, and acquires an inspection range image obtained by removing the non-inspection objects, and acquires a plurality of the inspection range images, and generates a normal model that represents a normal state in the inspection target location based on the plurality of inspection range images. (Supplementary Note 16) An anomaly detection program that causes a computer to execute the following steps: an inspection object detection process that acquires an inspection image obtained by capturing an area including an inspection target location that is the subject of anomaly detection, and detects the inspection target location from the inspection image; a moving object detection process that detects a moving object included in the inspection image; an outside-inspection removal process that removes, from the inspection image, an area other than the inspection target location and the moving object as outside-inspection objects, and acquires an inspection range image obtained by removing the outside-inspection objects; and an anomaly detection modeling process that acquires a plurality of the inspection range images, and generates a normal model that represents a normal state in the inspection target location based on the plurality of inspection range images.

[0089] 31 Inspection image, 32 Inspection range image, 40 Inspection target location, 41 Non-inspection object, 50 Normal model, 80 Signal line, 100 Anomaly detection device, 110 Pre-processing unit, 11 Inspection target detection unit, 111 White line detection unit, 112 Road area detection unit, 113 Method switching unit, 114 Moving object detection unit, 115 Frontal image generation unit, 116 Non-inspection removal unit, 117 Normalization unit, 120 Anomaly information detection unit, 121 Attribute classification unit, 122 Partial image classification unit, 123 Anomaly detection modeling unit, 124 Threshold application unit, 130 Information organization and display unit, 131 Anomaly information creation unit, 132 Anomaly information presentation unit, 909 Electronic circuit, 910 Processor, 921 Memory, 922 Auxiliary storage device, 930 Input / output interface, 950 Communication interface.

Claims

1. An anomaly detection device comprising: an inspection object detection unit that acquires an inspection image capturing an area including an inspection target location that is the subject of anomaly detection, and detects the inspection target location from the inspection image; a moving object detection unit that detects moving objects included in the inspection image; an outside-inspection removal unit that removes from the inspection image areas other than the inspection target location and the moving objects as outside-inspection objects, and acquires an inspection range image obtained by removing the outside-inspection objects; and an anomaly detection modeling unit that acquires a plurality of the inspection range images, and generates a normal model that represents a normal state in the inspection target location based on the plurality of inspection range images.

2. The anomaly detection device according to claim 1, wherein the anomaly detection modeling unit calculates the distance between the normal model and the inspection range image, and detects an anomaly in the inspection range image based on the distance.

3. The anomaly detection device according to claim 2, further comprising an attribute classification unit that classifies the inspection range images based on the attributes of the inspection target areas, and the anomaly detection modeling unit uses the multiple inspection range images to generate the normal model for each attribute of the inspection target areas.

4. The anomaly detection device according to claim 3, comprising: a partial image classification unit that classifies the inspection range image into a plurality of partial images based on the characteristics of the partial images included in the inspection range image; and a threshold application unit that sets a threshold value for the distance for each of the plurality of partial images, wherein the anomaly detection modeling unit calculates the distance between a partial image in a normal model that corresponds to the attribute of the inspection target area and a partial image in the inspection range image, and detects an abnormality in the partial image in the inspection range image using the distance and the threshold value corresponding to the partial image.

5. The anomaly detection device described in claim 4, wherein the inspection object detection unit acquires an image of the road surface as the inspection object location as the inspection image, and detects the road as the inspection object location from the inspection image by switching between a white line detection method that detects white lines on the road surface from the inspection image and a road area classification method that classifies the area of ​​the road from the inspection image.

6. The anomaly detection device described in claim 5, wherein the attribute classification unit classifies the inspection range images based on the road position information, and the anomaly detection modeling unit uses the multiple inspection range images to generate the normal model for each of the road position information.

7. An anomaly detection device as described in claim 5 or claim 6, wherein the partial image classification unit classifies the inspection range image into a plurality of partial images based on the attributes of the road contained in the inspection range image.

8. The anomaly detection device described in claim 4, wherein the inspection object detection unit acquires an image of a railway as the inspection object location as the inspection image, and detects the railway as the inspection object location from the inspection image by switching between a rail detection method that detects the railway rails from the inspection image and a railway area classification method that classifies the railway area from the inspection image.

9. The anomaly detection device described in claim 8, wherein the attribute classification unit classifies the inspection range images based on the railway location information, and the anomaly detection modeling unit uses the multiple inspection range images to generate the normal model for each of the railway location information.

10. An anomaly detection device as described in claim 8 or claim 9, wherein the partial image classification unit classifies the inspection range image into a plurality of partial images based on the characteristics of the railway contained in the inspection range image.

11. The anomaly detection device according to claim 4, wherein the inspection object detection unit acquires an image of an installation around the travel path as the inspection object location as the inspection image, and detects a part of the installation object as the inspection object location from the inspection image.

12. The anomaly detection device described in claim 11, wherein the attribute classification unit classifies the inspection range images based on the position information of the installed object, and the anomaly detection modeling unit uses the multiple inspection range images to generate the normal model for each position information of the installed object.

13. An anomaly detection device as described in claim 11 or claim 12, wherein the partial image classification unit classifies the inspection range image into a plurality of partial images based on the characteristics of the installation included in the inspection range image.

14. The anomaly detection device according to any one of claims 1 to 13, further comprising a front-facing image generation unit that converts the inspection image into a front-facing image viewed from a position directly facing the inspection target area.

15. An anomaly detection method in which a computer acquires an inspection image of an area including an inspection target location that is the target of anomaly detection, detects the inspection target location from the inspection image, detects a moving object included in the inspection image, removes from the inspection image an area other than the inspection target location and the moving object as non-inspection objects, and acquires an inspection range image obtained by removing the non-inspection objects, and acquires multiple inspection range images, and generates a normal model that represents the normal state of the inspection target location based on the multiple inspection range images.

16. An anomaly detection program that causes a computer to execute the following steps: an inspection object detection process that acquires an inspection image of an area including an inspection target location that is the target of anomaly detection, and detects the inspection target location from the inspection image; a moving object detection process that detects moving objects included in the inspection image; an outside-inspection removal process that removes from the inspection image areas other than the inspection target location and the moving objects as outside-inspection objects, and acquires an inspection range image obtained by removing the outside-inspection objects; and an anomaly detection modeling process that acquires multiple inspection range images and generates a normal model that represents the normal state of the inspection target location based on the multiple inspection range images.