Overhead wire fitting abnormality detection device and overhead wire fitting abnormality detection method

Abstract

To provide a line hardware abnormality detection apparatus and a line hardware abnormality detection method which effectively utilize a small amount of abnormal data acquired in the practical use by adopting a semi-supervised abnormality detection method utilizing normal data and a small amount of abnormal data.SOLUTION: A line hardware abnormality detection apparatus comprises: a camera 3 which is attached to the roof of a vehicle 2; a storage device 5 which stores image data output from the camera 3; a learning processing unit 9 which performs learning of an abnormality detection model with semi-supervised abnormality detection learning using one hot expression index image data obtained by converting the image data; and a data processing unit 7 which performs abnormality determination with the abnormality detection model.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an abnormality detection device and an abnormality detection method for an overhead line fitting. [Background technology]

[0002] In order to inspect railway facilities, cameras are mounted on vehicles, images of the railway facilities are captured, and the captured images are analyzed to detect abnormalities.

[0003] As an image analysis method, there is a method in which an anomaly detection model is created by machine learning and an anomaly is detected using this anomaly detection model. As a machine learning method, for example, one called DEEP SAD is disclosed as shown in Non-Patent Document 1. Moreover, Patent Document 1, which is an example of conventional technology, discloses a technology in which an image is generated using a generative model such as a neural network, the difference between the generated image and a captured image is calculated, and if this difference is large, an abnormality is determined. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Lukas Ruff, Robert A. Vandermeulen, Nico Gornitz, Alexander Binder, Emmanuel Muller, Klaus-Robert Muller, Marius Kloft, “Deep Semi-Supervised Anomaly Detection”, International Conference on Learning Representations 2020 (ICLR2020) [Patent documents]

[0005] [Patent Document 1] JP 2019-133306 A Summary of the Invention [Problem to be solved by the invention]

[0006] The anomaly judgment method described in Patent Document 1 judges anomalies based on the difference between an input image and a generated image of an image generator that has been trained using only normal data. However, there is a problem in that it is difficult to detect anomalies where the difference from the normal data is small. Anomaly detection methods using deep learning include supervised anomaly detection, semi-supervised anomaly detection, and unsupervised anomaly detection, and here we will describe the issues with supervised and unsupervised anomaly detection. In this specification, the method of learning using only normal data is referred to as unsupervised learning.

[0007] <Supervised learning> In railway equipment inspections, it is rare to obtain abnormal data, and most of the data that can be obtained is normal, so it is difficult to use supervised learning, which requires both normal and abnormal data. Even if supervised learning is used, if the ratio of the amount of normal data to the amount of abnormal data is not balanced, only normal data will be learned and anomaly detection will not work. In addition, it is necessary to label all data to be learned, which is very costly.

[0008] <Unsupervised learning> Unsupervised learning uses only normal data for learning, and data that deviates from normal conditions or normal data distribution is judged to be abnormal. However, since there is abnormal data with slight distortion or normal data that appears abnormal depending on the angle of view, it can be difficult to distinguish data near the boundary between normal and abnormal. Solving these problems by adding or modifying training data can also be difficult because there is no indicator of normal and abnormal. With unsupervised learning, there are cases where data that humans can identify as abnormal cannot be identified.

[0009] The present invention has been made in consideration of the above-mentioned situation, and the problem that the present invention aims to solve is to provide a wire fitting anomaly detection device and a wire fitting anomaly detection method that effectively utilize the small amount of abnormal data acquired during actual operation by adopting a semi-supervised anomaly detection technique that uses both normal data and a small amount of abnormal data. [Means for solving the problem]

[0010] In order to solve the above problems, the present invention employs the following means. The overhead line fitting anomaly detection device of the present invention comprises a camera mounted on the roof of a vehicle, a storage device for storing image data output from the camera, a learning processing unit that performs semi-supervised anomaly detection learning to learn an anomaly detection model using one-hot expression index image data converted from the image data, and a data processing unit that performs anomaly detection using the anomaly detection model. According to the present invention, an anomaly detection model is learned through semi-supervised anomaly detection learning, so that the acquired image data is effectively utilized to learn the model, and the model can be used to detect abnormalities in overhead line fittings.

[0011] In one aspect of the present invention, the learning processing unit includes a fitting detection unit that detects overhead line fittings from the image data, a segmentation unit that performs semantic segmentation on the image data of the detected overhead line fittings and converts the image data into one-hot expression index image data, a learning selection unit that selects either unsupervised anomaly detection learning, semi-supervised anomaly detection learning, or supervised anomaly detection learning based on the amount of abnormal data in the image data provided for model learning, an unsupervised learning unit that performs unsupervised anomaly detection learning, a semi-supervised learning unit that performs semi-supervised anomaly detection learning, and a supervised learning unit that performs supervised anomaly detection learning. In this aspect, either unsupervised anomaly detection learning, semi-supervised anomaly detection learning, or supervised anomaly detection learning is selected based on the amount of abnormal data in the image data provided for model learning, so that an appropriate learning method can be selected depending on the type of image data acquired.

[0012] In one aspect of the present invention, instead of performing the semantic segmentation by the segmentation unit, an annotation image is used when training a segmentation model that performs the semantic segmentation. In this embodiment, annotation images are used instead of performing semantic segmentation, so model learning can be performed efficiently.

[0013] The overhead line fitting anomaly detection method of the present invention includes taking images with a camera mounted on the roof of a vehicle, saving image data taken by the camera, learning an anomaly detection model by semi-supervised anomaly detection learning using one-hot expression index image data converted from the image data, and making an anomaly determination using the anomaly detection model. According to the present invention, an anomaly detection model is learned through semi-supervised anomaly detection learning, so that the acquired image data is effectively utilized to learn the model, and the model can be used to detect abnormalities in overhead line fittings.

[0014] In one aspect of the present invention, the learning content of the anomaly detection model includes detecting overhead line fittings from the image data, performing semantic segmentation on the image data of the detected overhead line fittings and converting the image data into one-hot expression index image data, and selecting and learning from unsupervised anomaly detection learning, semi-supervised anomaly detection learning, or supervised anomaly detection learning based on the amount of abnormal data in the image data provided for model learning. In this aspect, either unsupervised anomaly detection learning, semi-supervised anomaly detection learning, or supervised anomaly detection learning is selected based on the amount of abnormal data in the image data provided for model learning, so that an appropriate learning method can be selected depending on the type of image data acquired.

[0015] In one aspect of the invention, instead of performing semantic segmentation, the method includes providing annotation images for use in training a segmentation model that performs the semantic segmentation. In this embodiment, annotation images are used instead of performing semantic segmentation, so model learning can be performed efficiently. Effect of the Invention

[0016] According to the present invention, by adopting a semi-supervised anomaly detection technique that uses normal data and a small amount of abnormal data, it is possible to provide a wire fitting anomaly detection device and a wire fitting anomaly detection method that make effective use of the small amount of abnormal data acquired during actual operation. [Brief description of the drawings]

[0017] [Figure 1] 1 is a diagram showing an example of installation of a wire fitting abnormality detection device according to an embodiment of the present invention. FIG. [Diagram 2] 1 is a block diagram showing a configuration of a wire fitting abnormality detection device according to an embodiment of the present invention. [Diagram 3] 1 is a flowchart illustrating an anomaly detection process using a trained anomaly detection model according to an embodiment of the present invention. [Figure 4] 5A to 5C are diagrams showing 256-level, 3-channel RGB image data before being converted by a segmentation unit according to an embodiment of the present invention. [Diagram 5] FIG. 13 is a diagram showing one-hot expression index image data converted by a segmentation unit according to an embodiment of the present invention. [Figure 6] 4 is a flowchart illustrating a learning process of an anomaly detection model according to the embodiment of the present invention. [Figure 7] 1 is a diagram illustrating a concept of an anomaly detection method according to an embodiment of the present invention. [Figure 8] 4 is a flowchart illustrating a learning process of an anomaly detection model according to the embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0018] Hereinafter, an embodiment of the present invention will be described with reference to the accompanying drawings. (First embodiment) FIG. 1 is a diagram showing an example of installation of an overhead line fitting abnormality detection device according to an embodiment of the present invention. FIG. 2 is a block diagram showing a configuration of an overhead line fitting abnormality detection device according to an embodiment. As shown in FIG. 1, an overhead line fitting abnormality detection device 1 according to the present embodiment is installed on a vehicle 2 traveling on a rail R and is used to detect abnormalities in an overhead line K. As an example, the overhead line K may include a suspension wire K1 and a trolley wire K3 suspended from the suspension wire K1 by a hanger K2. The overhead line fitting abnormality detection device 1 includes a camera 3 attached to the roof of the vehicle 2, a storage device 5 for storing image data output from the camera 3, a learning processing unit 9 for learning an anomaly detection model by semi-supervised anomaly detection learning using one-hot expression index image data obtained by converting the image data, a data processing unit 7 for determining an abnormality by the anomaly detection model, and a UI (user interface) unit 11 for inputting and outputting settings of the overhead line fitting abnormality detection device 1 and detection results.

[0019] The camera 3 may be, for example, an area sensor camera or a line sensor camera. Although lighting or the like may be used if necessary, what is required is a means for photographing the linear metal fitting as in the monitoring image example shown in FIG. 1, and the configuration is not important. The storage device 5 may be, for example, a hard disk drive (HDD) or a solid state drive (SSD). The data processing unit 7 and the learning processing unit 9 may be configured by hardware, or each function may be realized by software. The data processing unit 7 and the learning processing unit 9 are, for example, information processing devices such as personal computers. Each of the functional blocks described later may be software or programs executed by a CPU or GPU in the information processing device. Alternatively, the data processing unit 7 and the learning processing unit 9 may be software or programs provided on a cloud service. The UI unit 11 may be, for example, a keyboard, a mouse, a touch panel, a display, etc. that perform input and output to and from a personal computer.

[0020] As shown in FIG. 2, the data processing unit 7 includes a fitting detection unit 71 that detects overhead line fittings from image data captured by the camera 3, a segmentation unit 72 that performs semantic segmentation on the image data of the detected overhead line fittings and converts the image data into one-hot expression index image data, and anomaly detection unit 73 that detects abnormalities in the overhead line fittings using an anomaly detection model obtained by anomaly detection learning.

[0021] The learning processing unit 9 includes a fitting detection unit 91 that detects overhead line fittings from image data, a segmentation unit 92 that performs semantic segmentation on the image data of the detected overhead line fittings and converts the image data into one-hot expression index image data, a learning selection unit 93 that selects either unsupervised anomaly detection learning, semi-supervised anomaly detection learning, or supervised anomaly detection learning based on the amount of abnormal data in the image data provided for model learning, an unsupervised learning unit 94 that performs unsupervised anomaly detection learning, a semi-supervised learning unit 95 that performs semi-supervised anomaly detection learning, and a supervised learning unit 96 that performs supervised anomaly detection learning.

[0022] Here, the metal fitting detection unit 71 of the data processing unit 7 and the metal fitting detection unit 91 of the learning processing unit 9 may be the same library executed on a CPU. The segmentation unit 72 of the data processing unit 7 and the segmentation unit 92 of the learning processing unit 9 may be the same library executed on a CPU.

[0023] <Overhead line fitting abnormality detection> FIG. 3 is a flowchart showing an anomaly detection process using a trained anomaly detection model 17 performed by the data processing unit 7. The operation of anomaly detection in this embodiment will be described with reference to this figure. The anomaly detection process is performed by the data processing unit 7 in FIG. 2. When the anomaly detection process is started (S01), the overhead wire K is continuously photographed by the camera 3 mounted on the vehicle (S02). The photographed image is sent to the storage device 5 and stored as image data (S03). The stored image data is sent to the metal fitting detection unit 71 of the data processing unit 7, and a metal fitting detection model 13 called from the storage device 5 is used to detect and extract the metal fittings that are the subject of equipment inspection from the photographed image data (S04). Any method may be used to detect the metal fittings. Examples of the detection method include simple image processing, pattern matching, and deep learning object detection methods.

[0024] The image data in which the metal fittings are detected and cut out by the metal fittings detection unit 71 is then sent to the segmentation unit 72. The segmentation unit 72 performs semantic segmentation on the cut out metal fitting image using the segmentation model 15 that has been previously learned and stored in the storage device 5 (S05). FIG. 4 shows RGB image data before performing semantic segmentation. The RGB image data has quantitative data of [0 to 255] in each of the three RGB channels. FIG. 5 shows index image data after performing semantic segmentation. The index image data has an index value as quantitative data of [0, 1] in each of a plurality of classes. By performing semantic segmentation in this way, the image shown in FIG. 4 is converted into one-hot representation and one-hot representation index image data shown in FIG. 5, thereby reducing the search space and reducing the learning difficulty of anomaly detection.

[0025] The image data that has been subjected to the semantic segmentation is sent to the anomaly detection unit 73. The anomaly detection unit 73 learns based on a learning flowchart described later, and detects anomalies using the anomaly detection model 17 stored in the storage device 5 (S06). Next, it is confirmed whether or not the image capture has ended (S07), and if the image capture continues, the image capture by the camera continues (S07: NO), and if the image capture has ended (S07: YES), the process ends (S08).

[0026] <Learning anomaly detection model> Fig. 6 is a flowchart showing the learning process of the anomaly detection model performed by the learning processing unit 9. Learning of the anomaly detection model in this embodiment will be described with reference to this figure. The metal fitting detection model 13 and segmentation model 15 used here may be the same as the models used for anomaly detection in Fig. 3. Learning of the anomaly detection model is performed by the learning processing unit 9 in Fig. 2. When learning of the anomaly detection model starts (S10), image data prepared for learning is sent from the storage device 5 to the metal fitting detection unit 91 of the learning processing unit 9 (S11).

[0027] The metal fitting detection unit 91 uses the metal fitting detection model 13 to detect and extract metal fittings that are the subject of equipment inspection from the received image data (S12). The metal fitting detection process here may be the same process as the metal fitting detection (S04) in FIG. 3, and may be prepared as a library executed by the CPU as described above. The image data after the metal fitting detection process in the metal fitting detection unit 91 has been completed is sent to the segmentation unit 92.

[0028] The segmentation unit 92 performs semantic segmentation on the cut-out metal fitting image using the segmentation model 15 that has been trained in advance (S13). The semantic segmentation process here may be the same process as the segmentation (S05) in FIG. 3, and may be prepared as a library executed by the CPU as described above. The image data on which the semantic segmentation has been performed in the segmentation unit 92 is sent to the learning selection unit 93.

[0029] The learning selection unit 93 receives the sent image data, and after predetermined learning image data is accumulated, checks whether or not there is an index of abnormal data as teacher data for this image data. The learning selection unit 93 selects a learning method for the anomaly detection model based on the amount of abnormal data confirmed (S14). Here, an index (flag) of abnormal data or normal data is attached to any image data after metal fitting detection (S12).

[0030] If the image data processed and accumulated up to this point does not contain any abnormal data, it is sent to unsupervised learning unit 94, where unsupervised anomaly detection learning is performed (S15). If the amount of abnormal data is less than a predetermined threshold, the image data is sent to semi-supervised learning unit 95, where semi-supervised anomaly detection learning is performed (S16). If the amount of abnormal data is abnormal to a predetermined threshold, the image data is sent to supervised learning unit 96, where supervised anomaly detection learning is performed (S17).

[0031] Here, examples of unsupervised anomaly detection learning methods include VAE (Variational Autoencoder) and AnoGAN (Anomaly Generative Adversarial Network). Examples of semi-supervised anomaly detection learning methods include Deep SAD (Deep Semi-supervised Anomaly Detection) and FCDD (Fully Convolutional Data Description), which will be described later. Examples of supervised anomaly detection learning methods include two-class classification using SVM (Support Vector Machine) and CNN (Convolutional Neural Networks) such as ResNet.

[0032] The anomaly detection model generated by model learning processing in either the unsupervised learning unit 94, the semi-supervised learning unit 95, or the supervised learning unit 96 is stored in the storage device 5 (S18), and the series of anomaly detection model learning processes is completed (S19).

[0033] <Semi-supervised anomaly detection learning> In the semi-supervised anomaly detection learning in this embodiment, learning is performed on one-hot expression index image data converted into one-hot expression. Any semi-supervised learning method may be used as long as it can use normal data and a small amount of abnormal data. Here, the case where the semi-supervised anomaly detection method Deep SAD (Deep Semi-supervised Anomaly Detection) described in Non-Patent Document 1 is used is shown.

[0034] Figure 7 is a conceptual diagram showing the model learning structure in Deep SAD. In Deep SAD learning, the one-hot representation index image data D input is converted into a one-hot representation, and feature values ​​T are extracted by an Encoder (E), and the results of this feature extraction are subjected to distance learning in the latent variable space G. Deep SAD learning is performed by minimizing the objective function shown in the following [Equation 1].

[0035]

number

[0036] Here, x is the input data, y is the label information (normal = 1 or abnormal = -1), n ​​is the number of unlabeled data, m is the number of labeled data, η is a hyperparameter that weights the learning of labeled data and unlabeled data, φ is the output of the Encoder, W is the weight parameter of the network, and c is the center of the hypersphere calculated from the learning data. In [Equation 1], the first term is the calculation for unlabeled data, the second term is the calculation for labeled data, and the third term is the weighting factor for normalization. As can be seen from the objective function, Deep SAD can use all data for learning, including normal data, a small amount of abnormal data, and unlabeled data. In anomaly detection using Deep SAD, the distance from the center c of the hypersphere shown in [Equation 2] below is calculated as the anomaly score, and anomaly detection is performed by setting a threshold for the anomaly score.

[0037]

number

[0038] The degree of abnormality (score) is small if normal and large if abnormal, so if it is smaller than a set threshold it is judged to be normal, and if it is larger it is judged to be abnormal.

[0039] As described above, in this embodiment, even if the image data for model learning has little abnormal data, an anomaly detection model can be created by performing semi-supervised anomaly detection learning to detect an anomaly. Therefore, all of the normal data, abnormal data, and data in which normal and abnormal data are not distinguished can be utilized, and the cost of model learning can be reduced. Since it is based on unsupervised learning that assumes the distribution of normal data, normal data and abnormal data can be separated even in a data set in which normal data and abnormal data are unbalanced. Therefore, effective learning can be performed on an unbalanced data set.

[0040] In the semi-supervised learning of this embodiment, data that is difficult to separate in unsupervised learning can be explicitly separated by the supervised learning part of the semi-supervised learning. Therefore, it is possible to improve the accuracy of anomaly detection by utilizing a small amount of abnormal data. When learning a model, an index image that expresses image data as one hot is generated to keep the search space small, and learning processing and anomaly detection processing are facilitated, making stable processing possible. In addition, this embodiment can be effectively used in the transitional period of data acquisition in actual operation, and by adding abnormal data acquired in actual operation to learning, it is possible to improve the anomaly detection device to a more robust one. Furthermore, based on the amount of abnormal data in the data for learning the anomaly detection model, any of unsupervised anomaly detection learning, semi-supervised anomaly detection learning, and supervised anomaly detection learning can be selected, so that the optimal learning method can be selected according to the characteristics of the data for learning the model.

[0041] Second embodiment FIG. 8 is a flowchart of the learning process of the anomaly detection model in this embodiment. This embodiment differs from the first embodiment in that the process S13 for performing semantic segmentation in FIG. 6 is changed to a process S23 for using the annotation image used when learning the segmentation model. The other processes are similar to each other, so the same reference numerals are used and the description thereof is omitted. In this embodiment, in addition to the same effects as the first embodiment, the annotation image used when learning the segmentation model in the past is supplied and used when learning the model, so that the learning process steps can be shortened and the anomaly detection model can be learned efficiently. [Explanation of symbols]

[0042] 1. Overhead wire fitting abnormality detection device 3. Camera 5 Storage device 7 Data Processing Section 9 Learning Processing Section 91 Metal fitting detector 92 Segmentation Department 93 Learning Selection Section 94 Unsupervised Learning 95 Semi-supervised learning section 96 Supervised Learning

Claims

1. A camera mounted on the roof of the vehicle; a storage device for storing image data output from the camera; a learning processing unit that learns an anomaly detection model by semi-supervised anomaly detection learning using one-hot expression index image data obtained by converting the image data; A data processing unit that performs an abnormality determination using the anomaly detection model, The learning processing unit: A metal fitting detection unit that detects wire fittings from the image data; a segmentation unit that performs semantic segmentation on the image data of the detected overhead line fittings and converts the image data into one-hot expression index image data; A learning selection unit that selects one of unsupervised anomaly detection learning, semi-supervised anomaly detection learning, and supervised anomaly detection learning based on the amount of abnormal data in the image data provided for model learning; An unsupervised learning unit that performs unsupervised anomaly detection learning; A semi-supervised learning unit that performs semi-supervised anomaly detection learning; An overhead line fitting anomaly detection device comprising: a supervised learning unit that performs supervised anomaly detection learning.

2. The overhead line fitting abnormality detection device according to claim 1 , wherein an annotation image used for learning a segmentation model for performing the semantic segmentation is used instead of performing the semantic segmentation by the segmentation unit.

3. Shooting with a camera mounted on the roof of the vehicle; storing image data captured by the camera; learning an anomaly detection model by semi-supervised anomaly detection learning using one-hot expression index image data obtained by converting the image data; performing an anomaly determination using the anomaly detection model; The learning content of the anomaly detection model is as follows: Detecting overhead line fittings from the image data; performing semantic segmentation on the image data of the detected overhead line fittings and converting the image data into one-hot expression index image data; A method for detecting abnormalities in overhead line fittings, comprising selecting and performing learning from unsupervised anomaly detection learning, semi-supervised anomaly detection learning, or supervised anomaly detection learning based on the amount of abnormal data in image data provided for model learning.

4. The overhead line fitting anomaly detection method according to claim 3, further comprising, instead of performing semantic segmentation, supplying annotation images to be used when learning a segmentation model that performs the semantic segmentation.

Images