Overhead wire fitting abnormality detection device and overhead wire fitting abnormality detection method

Abstract

Provided is a line fitting abnormality detection device and method that can improve the detection accuracy of abnormalities in line fittings at low cost even when the parts to be detected are small. The device includes a learning processing unit 17 that performs learning using image data generated by photographing the line fittings, and a data processing unit 15 that performs abnormality judgment using the image data, the learning processing unit 17 includes an equipment learning unit 19 that learns to detect equipment, and a part learning unit 21 that learns to detect parts of the equipment, the data processing unit 15 detects the equipment using an equipment detection model learned by the equipment learning unit 19, detects the parts from the detected equipment using a part detection model learned by the part learning unit 21, and judges abnormalities in the equipment and / or parts using a previously learned anomaly detection model.

Description

[Technical field]

[0001] The present invention relates to an abnormality detection device and an abnormality detection method for an overhead contact fitting. [Background technology]

[0002] Conventionally, maintenance and inspection work has been carried out on overhead line fittings, which are part of electric railway equipment, and inspectors visually check for abnormalities in the overhead line fittings to perform maintenance. Overhead line fittings are fittings that hold the overhead train wires (trolley wires, suspension wires, and auxiliary suspension wires; hereafter referred to as "overhead wires") in place to supply power to trains in electric railway equipment, or electrically connect or separate multiple overhead wires. However, visual inspection has limitations and there is a risk of overlooking abnormalities, so there has been a demand for automating this inspection.

[0003] Therefore, Patent Document 1 discloses a technology in which an object is photographed with a camera mounted on a moving object, the photographed image is used to generate an image using a generative model such as a neural network, the similarity between the generated image and the photographed image is calculated, and if this similarity is small, an abnormality is determined.

[0004] Patent Document 2 discloses an anomaly detection device for overhead wire fittings that performs learning for each part constituting the overhead wire fittings and detects anomalies in order to improve the accuracy of anomaly detection. In this device, first, an image of the sky above the railway vehicle is acquired by a line sensor camera mounted on the roof of the railway vehicle. Next, from the acquired image, the train line fittings that have been learned by machine learning object detection are detected, and semantic segmentation is applied to the target equipment image to create a one-hot representation. This simplifies the image and makes it easier to judge anomalies in shape. Since it is possible to extract only the areas of each part of the fittings by semantic segmentation, a technology is disclosed in which this is used to generate a mask image of only the part parts, normal data is learned for each part, and anomalies are judged using the restoration error of the masked part. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2019-133306 A [Patent Document 2] Patent Publication No. 2022-159010 Summary of the Invention [Problem to be solved by the invention]

[0006] However, the technology disclosed in Patent Document 1 does not perform detection over a small area, so if the part to be detected is small, it may be difficult to improve detection accuracy.

[0007] The technology disclosed in Patent Document 2 requires semantic segmentation, which has the problem of high annotation and learning costs. Another problem is that the accuracy (error) of segmentation affects the subsequent anomaly determination. In addition, the target equipment detected during anomaly detection needs to be resized so that it can be handled by the anomaly detection model, but if the target part is small compared to the target equipment, information is lost due to resizing, making it difficult to determine an anomaly.

[0008] Therefore, an object of the present invention is to provide a wire fitting abnormality detection device and a wire fitting abnormality detection method that can improve the accuracy of detecting abnormalities in wire fittings at low cost even when the parts to be detected are small. [Means for solving the problem]

[0009] In order to solve the above-mentioned problems, the overhead line fitting abnormality detection device of the present invention comprises a learning processing unit that performs learning using image data generated by photographing the overhead line fittings, and a data processing unit that performs abnormality judgment using the image data, wherein the learning processing unit comprises an equipment learning unit that learns to detect equipment, and a part learning unit that learns to detect parts of the equipment, and the data processing unit detects the equipment using an equipment detection model learned by the equipment learning unit, detects the parts from the detected equipment using a part detection model learned by the part learning unit, and judges abnormalities in the equipment and / or parts using a previously trained anomaly detection model.

[0010] In one aspect of the present invention, the data processing unit detects the facility based on the facility detection model, vehicle traveling position information, and facility information.

[0011] In one aspect of the present invention, the data processing unit determines an abnormality in equipment and / or components based on a restoration error of the anomaly detection model.

[0012] In one aspect of the present invention, the system further includes an input / output unit, and the detection results of the equipment and / or parts detected by the data processing unit are displayed on the input / output unit, and an operator can check and edit the detection results from the input / output unit.

[0013] The overhead line fitting abnormality detection method of the present invention includes performing learning using image data generated by photographing the overhead line fittings, and performing abnormality judgment using the image data, wherein the performing learning includes learning to detect equipment, and learning to detect parts of the equipment, and performing abnormality judgment includes detecting the equipment using an equipment detection model learned by learning to detect the equipment, detecting the parts from the detected equipment using a part detection model learned by learning to detect the parts of the equipment, and judging abnormalities in the equipment and / or parts using a previously trained anomaly detection model.

[0014] In one aspect of the present invention, performing the abnormality determination includes detecting the facility based on the facility detection model, vehicle traveling position information, and facility information.

[0015] In one aspect of the present invention, performing the anomaly determination includes determining an anomaly in equipment and / or components based on a recovery error of the anomaly detection model.

[0016] One aspect of the present invention includes displaying the detection results of the equipment and / or the parts detected by performing the abnormality judgment on an input / output unit, and an operator checking and editing the detection results from the input / output unit.

[0017] A catenary fitting abnormality detection device according to one aspect of the present invention comprises a learning processing unit that performs learning using image data generated by photographing catenary fittings, and a data processing unit that performs abnormality judgment using the image data, wherein the learning processing unit comprises an equipment learning unit that learns to detect equipment, and a part learning unit that learns to detect parts of the equipment, and the data processing unit detects the equipment using an equipment detection model learned by the equipment learning unit, detects the parts from the detected equipment using a part detection model learned by the part learning unit, corrects the image data detecting the equipment or parts, and judges abnormalities in the equipment and / or parts using a previously learned anomaly detection model.

[0018] According to another aspect of the present invention, an overhead line fitting abnormality detection device comprises a learning processing unit that performs learning using image data generated by photographing overhead line fittings, and a data processing unit that performs abnormality judgment using the image data, wherein the learning processing unit comprises an equipment learning unit that learns to detect equipment, and a part learning unit that learns to detect parts of the equipment, and the data processing unit corrects the image data, detects the equipment using an equipment detection model learned by the equipment learning unit, detects the parts from the detected equipment using a part detection model learned by the part learning unit, and judges abnormalities in the equipment and / or parts using a previously trained anomaly detection model.

[0019] According to yet another aspect of the present invention, an overhead line fitting abnormality detection device comprises a learning processing unit that performs learning using image data generated by photographing the overhead line fittings, and a data processing unit that performs abnormality judgment using the image data, wherein the learning processing unit comprises an equipment learning unit that learns to detect equipment, and a part learning unit that learns to detect parts of the equipment, and the data processing unit detects the equipment using an equipment detection model learned by the equipment learning unit, detects the parts from the detected equipment using a part detection model learned by the part learning unit, corrects the image data detecting the equipment or parts based on the photographing time and vehicle traveling position information, and judges abnormalities in the equipment and / or parts using a previously trained anomaly detection model.

[0020] According to yet another aspect of the present invention, a catenary fitting abnormality detection device comprises a learning processing unit that performs learning using image data generated by photographing catenary fittings, and a data processing unit that performs abnormality judgment using the image data, wherein the learning processing unit comprises an equipment learning unit that learns to detect equipment, and a part learning unit that learns to detect parts of the equipment, and the data processing unit detects the equipment using an equipment detection model learned by the equipment learning unit, detects the parts from the detected equipment using a part detection model learned by the part learning unit, corrects the image data based on luminance information or an identification AI of the image data detecting the equipment or parts, and judges abnormalities in the equipment and / or parts using a pre-trained anomaly detection model.

[0021] In one aspect of the present invention, the data processing unit performs an inversion process on the image data based on the position of the detected object in the image data before determining an abnormality in the equipment and / or parts using the pre-trained anomaly detection model.

[0022] A method for detecting abnormalities in overhead line fittings according to one aspect of the present invention includes learning using image data generated by photographing overhead line fittings and making an abnormality judgment using the image data, where the learning includes learning to detect equipment and learning to detect parts of the equipment, and where making the abnormality judgment includes detecting the equipment using an equipment detection model learned by learning to detect the equipment, detecting the parts from the detected equipment using a part detection model learned by learning to detect the parts of the equipment, correcting the image data in which the equipment or the parts are detected, and judging abnormalities in the equipment and / or parts using a previously trained anomaly detection model.

[0023] A method for detecting abnormalities in overhead line fittings according to another aspect of the present invention includes learning using image data generated by photographing overhead line fittings, and making an abnormality judgment using the image data, wherein the learning includes learning to detect equipment, and learning to detect parts of the equipment, and the making an abnormality judgment includes correcting the image data, detecting the equipment using an equipment detection model learned by learning to detect the equipment, detecting the parts from the detected equipment using a part detection model learned by learning to detect the parts of the equipment, and judging abnormalities in the equipment and / or parts using a pre-trained anomaly detection model.

[0024] A method for detecting an abnormality in overhead line fittings according to yet another aspect of the present invention includes performing learning using image data generated by photographing the overhead line fittings, and performing an abnormality judgment using the image data, wherein the performing learning includes learning to detect equipment, and learning to detect parts of the equipment, and performing the abnormality judgment includes detecting the equipment using an equipment detection model learned by learning to detect the equipment, detecting the parts from the detected equipment using a part detection model learned by learning to detect the parts of the equipment, correcting the image data detecting the equipment or the parts based on the photographing time and vehicle traveling position information, and judging an abnormality in the equipment and / or the parts using a pre-trained anomaly detection model.

[0025] A method for detecting an abnormality in overhead line fittings according to yet another aspect of the present invention includes learning using image data generated by photographing overhead line fittings, and making an abnormality judgment using the image data, where the learning includes learning to detect equipment, and learning to detect parts of the equipment, and where making an abnormality judgment includes detecting the equipment using an equipment detection model learned by learning to detect the equipment, detecting the parts from the detected equipment using a part detection model learned by learning to detect the parts of the equipment, correcting the image data based on luminance information or an identification AI of the image data detecting the equipment or the parts, and judging an abnormality in the equipment and / or parts using a pre-trained anomaly detection model.

[0026] In one aspect of the present invention, performing the anomaly judgment includes performing an inversion process of the image data based on the position of a detected object in the image data before judging an abnormality in equipment and / or parts using the pre-trained anomaly detection model. Effect of the Invention

[0027] According to the present invention, it is possible to provide a wire fitting abnormality detection device and a wire fitting abnormality detection method that can improve the accuracy of detecting abnormalities in the wire fitting at low cost even when the parts to be detected are small. [Brief description of the drawings]

[0028] [Figure 1] 1 is a schematic diagram showing an example of installation of a wire fitting abnormality detection device according to an embodiment of the present invention. [Diagram 2] 1 is a block diagram showing a configuration of a wire fitting detection device according to an embodiment of the present invention. [Diagram 3] 4 is a flowchart illustrating the operation of the overhead line fitting abnormality detection device according to the embodiment of the present invention. [Figure 4] 10 is a flowchart illustrating an operation of facility detection learning according to an embodiment of the present invention. [Diagram 5] 10A and 10B are schematic diagrams illustrating facility detection results according to an embodiment of the present invention. [Figure 6] 4 is a flowchart illustrating an operation of part detection learning according to an embodiment of the present invention. [Figure 7] 10A and 10B are schematic diagrams illustrating component detection results according to an embodiment of the present invention. [Figure 8] 1A to 1C are schematic diagrams illustrating image cropping and resizing results according to an embodiment of the present invention. [Figure 9] 4 is a flowchart illustrating an anomaly detection learning operation according to an embodiment of the present invention. [Figure 10] 10A and 10B are schematic diagrams illustrating an abnormality detection result according to an embodiment of the present invention. [Figure 11] 4 is a flowchart illustrating the operation of the overhead line fitting abnormality detection device according to the embodiment of the present invention. [Figure 12] 4 is a flowchart illustrating the operation of the overhead line fitting abnormality detection device according to the embodiment of the present invention. [Figure 13] 1 is a schematic diagram illustrating anomaly detection based on a restoration error according to an embodiment of the present invention; [Figure 14]4 is a flowchart illustrating the operation of the overhead line fitting abnormality detection device according to the embodiment of the present invention. [Figure 15] 4 is a flowchart illustrating the operation of the overhead line fitting abnormality detection device according to the embodiment of the present invention. [Figure 16] 4 is a flowchart illustrating the operation of the overhead line fitting abnormality detection device according to the embodiment of the present invention. [Figure 17] 4 is a flowchart illustrating the operation of the overhead line fitting abnormality detection device according to the embodiment of the present invention. [Figure 18] 4 is a flowchart illustrating the operation of the overhead line fitting abnormality detection device according to the embodiment of the present invention. [Figure 19] 4 is a flowchart illustrating the operation of the overhead line fitting abnormality detection device according to the embodiment of the present invention. [Figure 20] 4 is a schematic diagram showing the positional relationship between a camera and a wire fitting, for explaining the image inversion process of the wire fitting abnormality detection device according to the embodiment of the present invention. FIG. [Figure 21] 5 is a schematic diagram for explaining an image inversion process of the overhead line fitting abnormality detection device according to the embodiment of the present invention. FIG. [Figure 22] 5 is a schematic diagram for explaining an image inversion process of the overhead line fitting abnormality detection device according to the embodiment of the present invention. FIG. [Figure 23] 5 is a schematic diagram for explaining an image inversion process of the overhead line fitting abnormality detection device according to the embodiment of the present invention. FIG. [Figure 24] 5 is a schematic diagram for explaining an image inversion process of the overhead line fitting abnormality detection device according to the embodiment of the present invention. FIG. [Diagram 25] 1 is a schematic diagram showing a relationship between a railway track and an overhead wire, for explaining an image inversion process of an overhead wire fitting abnormality detection device according to an embodiment of the present invention; FIG. [Figure 26] 4 is a flowchart illustrating the operation of the overhead line fitting abnormality detection device according to the embodiment of the present invention. [Figure 27] 4 is a flowchart illustrating the operation of the overhead line fitting abnormality detection device according to the embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0029] Hereinafter, an embodiment of the present invention will be described with reference to the accompanying drawings. A. First embodiment Fig. 1 is a schematic diagram showing an example of installation of an overhead line fitting abnormality detection device 13 according to a first embodiment of the present invention. Fig. 1 shows a vehicle 1 which is a railway vehicle, a railway track 2 on which the vehicle 1 runs, and an overhead line fitting 3 which is an object of abnormality detection. The vehicle 1 shown in Fig. 1 includes a camera 11, a storage device 12, an overhead line fitting abnormality detection device 13, and a user interface (input / output unit) 14.

[0030] The camera 11 is installed on the roof of the vehicle 1, photographs the overhead line fittings 3, and outputs image data. Examples of the camera 11 include an area sensor camera and a line sensor camera. Note that, for photographing with the camera 11, a lighting device (not shown) may be provided as necessary to project light onto the overhead line fittings 3.

[0031] The storage device 12 stores image data output from the camera 11. The user interface 14 includes an input unit including at least one of a pointing device and a keyboard, and an output unit including a display. Here, the storage device 12 may be implemented by a storage device such as a semiconductor memory or a magnetic disk.

[0032] FIG. 2 is a block diagram showing the configuration of the overhead line fitting abnormality detection device 13 according to this embodiment. The overhead line fitting abnormality detection device 13 includes a data processing unit 15 having object detection and abnormality judgment functions, and a learning processing unit 17 that learns an object detection and abnormality judgment model, and detects abnormalities in the overhead line fittings 3 based on image data stored in the storage device 12. The learning processing unit 17 includes an equipment learning unit 19 that learns to detect equipment, and a part learning unit 21 that learns to detect parts of the equipment. Based on the image data stored in the storage device 12, the equipment learning unit 19 generates an equipment detection model and the part learning unit 21 generates a part detection model by deep learning. The data processing unit detects equipment using the equipment detection model learned by the equipment learning unit, detects parts from the detected equipment using the part detection model learned by the part learning unit, and judges abnormalities in the equipment and / or parts using the abnormality detection model learned in advance. The overhead line fitting abnormality detection device 13 may be configured with an information processing device such as a personal computer.

[0033] 3 is a flowchart for explaining the operation of the overhead line fitting abnormality detection device 13 according to this embodiment. The operation of the overhead line fitting abnormality detection device 13 will be explained using this flowchart.

[0034] <Image input> The overhead line fittings 3 are photographed by a camera 11 attached to the roof of a vehicle 1 (FIG. 3, S001). <Save to storage device> The image data captured by the camera 11 is stored in the storage device 12 (FIG. 3, S002).

[0035] <Facility detection> The target equipment is detected using the equipment detection model from the equipment detection model database M01 (FIG. 3, S003). The equipment detection model has been subjected to deep learning in advance. FIG. 4 is a flowchart showing the learning operation of the equipment detection model executed by the equipment learning unit 19. FIG. 5 is a schematic diagram showing the result of detecting the target equipment using the equipment detection model MS. As shown in FIG. 4, the equipment learning unit 19 first reads image data from the traveling image dataset D03 (FIG. 4, S101). Next, equipment detection learning is performed using data including both position information (coordinates of the four vertices of the rectangular area) surrounding the target equipment of the image data from the equipment annotation database D04 and a correct answer label (FIG. 4, S102). The equipment detection model obtained by deep learning is stored in the equipment detection model database M01 (FIG. 4, S103).

[0036] Fig. 5 illustrates an example of the operation of detecting equipment using the equipment detection model MS that has been trained in advance by deep learning. The equipment detection model MS receives an input image G1 and outputs an equipment detection result G2. As the equipment detection result, "rectangular coordinate information of the equipment" is acquired as "rectangular coordinate information in the image data" indicated by the dashed line A in the equipment detection result G2, and an "equipment label" corresponding to the type of equipment is acquired as "label information." <Equipment detection result judgment> Next, if the equipment is detected, the process proceeds to a part presence determination (FIG. 3, S004, YES).If the equipment is not detected, the process proceeds to a shooting end condition determination (FIG. 3, S004, NO).

[0037] <Part presence determination> When equipment is detected by equipment detection (Fig. 3, S003), a determination is made as to whether the detected equipment has a target part based on the "equipment label" and the inspection item database D01 (Fig. 3, S005). Here, the inspection item database D01 records whether the target part exists in each equipment label. If the target part exists, the process proceeds to part detection (Fig. 3, S005, YES), and if it does not exist, the process proceeds to image cropping and resizing (Fig. 3, S005, NO).

[0038] <Part detection> In part detection, object detection is performed within a rectangular area detected by equipment detection (FIG. 3, S003). At this time, part detection is performed using a part detection model from the part detection model database M02 for each target equipment in which a target part exists (FIG. 3, S006). The part detection model has been subjected to deep learning in advance. FIG. 6 is a flowchart showing the learning operation of the part detection model executed by the part learning unit 21. FIG. 7 is a schematic diagram showing the result of detecting a target part using the part detection model MB. As shown in FIG. 6, the part learning unit 21 first reads image data from the equipment image data set D05 (FIG. 6, S201). Next, part detection learning is performed using data including both position information (coordinates of the four vertices of the rectangular area) surrounding the target part of the image data from the part annotation database D06 and a correct answer label (FIG. 6, S202). The deep-learned part detection model is stored in the part detection model database M02 (FIG. 6, S203). Next, if learning of all the equipment parts is not completed (FIG. 6, S204, NO), the same flow is repeated again from image input S201. If learning of all the equipment parts is completed (FIG. 6, S204, YES), the learning operation ends.

[0039] FIG. 7 illustrates an example of equipment detection using the part detection model MB that has been trained in advance by deep learning. The part detection model MB receives input image G3 and outputs part detection result G4. As part detection results, multiple "rectangular coordinate information of parts" indicated by dashed line B in detection result G2 as "rectangular coordinate information in image data" and "part labels" corresponding to the type of parts as "label information" are acquired. Here, as detection results, the detection results of both equipment and parts are retained, such as the "rectangular coordinate information of equipment" and "equipment label" carried over from equipment detection, and the "rectangular coordinate information of parts" and "part labels" detected in part detection.

[0040] <Image cropping and resizing> An image is cut out from the image data based on the "rectangular coordinate information of the equipment" indicated by dashed line A in Fig. 5, or the "rectangular coordinate information of the part" indicated by dashed line B in Fig. 7, and resized to an image size that can be handled by the anomaly detection model (Fig. 3, S007). Fig. 8 is a schematic diagram showing an example of the results of image cut-out and resizing. Fig. 8 shows an example of image cut-out and resizing for the rectangular coordinate area of ​​dashed line A for the equipment detection result. The cut-out image is resized to a square for object detection. The "rectangular coordinate information in the image data," "label information," and "resized cut-out image" are sent to subsequent processing.

[0041] <Abnormality degree calculation> The anomaly detection model of each equipment or part that has been previously learned is read from the anomaly detection model database M03, and is applied to the resized cut-out image to calculate the degree of anomaly (FIG. 3, S008). The anomaly detection model has been previously deep-learned. FIG. 9 is a flowchart showing the learning operation of the anomaly detection model executed by the equipment learning unit 19 and the part learning unit 21. FIG. 10 is a schematic diagram showing the result of anomaly degree calculation using the anomaly detection model MK. As shown in FIG. 9, the equipment learning unit 19 and the part learning unit 21 first read image data from the respective anomaly detection image data sets D07 (FIG. 9, S301). Anomaly detection learning is performed from the read image (FIG. 9, S302). The deep-learned anomaly detection model is stored in the anomaly detection model database M03 (FIG. 9, S303). Next, if anomaly detection learning has not been completed for all equipment and parts (FIG. 9, S304, NO), the same flow is repeated again from image input S301. When abnormality detection learning for all equipment and components is completed (FIG. 9, S304, YES), the learning operation ends.

[0042] FIG. 10 illustrates an example of an anomaly detection operation using an anomaly detection model MK that has been trained in advance by deep learning. The anomaly detection model MK receives input images G5 and G6 and outputs a part detection result G7. Here, G5 is an example of normal data, and G6 is an example of abnormal data. In this embodiment, a distance-based anomaly detection model using multidimensional features is illustrated as an anomaly detection model. The anomaly detection model used here may be supervised or unsupervised as learning data as long as it outputs multidimensional features, but in the case of supervised learning data, annotation of normal or abnormal is required in advance. The output multidimensional features are plotted in a latent variable space, and the degree of anomaly is calculated based on the distance from the normal center. Abnormal data is plotted at a position relatively far from the normal center.

[0043] <Abnormality judgment> The degree of anomaly calculated as a distance in the latent variable space is read from the anomaly detection threshold database D02 created in advance to read the anomaly judgment threshold set for each target equipment and target part, and an anomaly judgment is performed using the threshold (Fig. 3, S009). As described above, when the output is a multidimensional feature, the degree of anomaly is calculated as the distance from the normal center, and if the degree of anomaly is equal to or greater than the threshold (Fig. 3, S009, YES), an anomaly flag is assigned (Fig. 3, S010). If the degree of anomaly is not equal to or greater than the threshold (Fig. 3, S009, NO), a normal flag is assigned (Fig. 3, S011). After each flag is assigned, the anomaly detection result is saved (Fig. 3, S012). The information saved here is "rectangle coordinate information", "label information", and "anomaly judgment result". <End of shooting determination> Finally, it is determined whether processing has been completed for all images (FIG. 3, S013). If shooting has not ended (FIG. 3, S013, NO), the series of flows is repeated from image input S001. If shooting has ended (FIG. 3, S013, YES), the processing ends.

[0044] In this processing flow, examples of deep learning techniques using detection models for equipment detection (FIG. 3, S004) and part detection (FIG. 3, S006) include, but are not limited to, SSD (Single Shot Multibox Detector).

[0045] As described above, in this embodiment, equipment detection is performed using the equipment detection model, and for the detected equipment, part detection is performed using the part detection model for each equipment. Since the degree of anomaly is calculated for each detected equipment and part using the respective anomaly detection model, anomaly detection can be performed not only for equipment in a relatively large area but also for parts in a small area, and the accuracy of anomaly detection can be improved. By performing object detection in stages, from equipment detection to part detection, it is possible to detect a relatively small area without using semantic segmentation, and by adopting object detection with low calculation cost and annotation cost for target part detection, it is possible to reduce the cost of calculation and improve the detection accuracy. Thus, it is possible to provide an overhead line fitting abnormality detection device and an overhead line fitting abnormality detection method that can improve the detection accuracy of an abnormality in an overhead line fitting at low cost even when the detection target part is small.

[0046] B. Second embodiment FIG. 11 shows a flowchart of the operation of the overhead line fitting abnormality detection device 13 of this embodiment. This embodiment differs from the first embodiment in that equipment installation location information (equipment information) and vehicle running location information F2 from the equipment database F1 are added to equipment detection, and equipment installation location information and vehicle running location information F4 from the equipment database F3 are added to part detection. Other than this, the operations and functions are the same, so the same reference numerals are given in the drawings and their explanations are omitted. Adding vehicle running information and equipment installation information to equipment detection and part detection means that, for example, it is possible to determine whether equipment is installed from vehicle location information of a captured image. As a result, in addition to the effects of the first embodiment, overdetection in which equipment or parts are determined to exist when they do not exist can be suppressed, and detection at a running position where the equipment or parts are installed can be ensured, thereby improving detection accuracy.

[0047] C. Third embodiment FIG. 12 shows a flowchart of the operation of the overhead wire fitting abnormality detection device 13 of this embodiment. FIG. 13 shows a schematic diagram showing an abnormality detection operation using the abnormality detection model MA based on the restoration error in this embodiment. This embodiment is different from the first embodiment in that the restoration error calculated from the image restored by an autoencoder or the like and the input image is used for the abnormality degree judgment (FIG. 12, S409). Other than this, the operation and function are the same, so the same reference numerals are given in the drawings and the description is omitted. As shown in FIG. 13, the abnormality detection model MA in this embodiment is composed of an encoder and a decoder, and inputs images G8 and G9 and outputs restored images G10 and G11. Then, the difference between the input images G8 and G9 and the restored images G10 and G11 is compared and the result is output. For the difference comparison, for example, the difference between the RGB values ​​for each pixel is calculated, and the sum of these is used as an index of the abnormality degree. Therefore, in addition to the action and effect of the first embodiment, the abnormality degree calculation method of this embodiment can be applied based on the detection target to be implemented, and the detection accuracy can be improved.

[0048] D. Fourth embodiment FIG. 14 shows a flowchart of the operation of the overhead line fitting abnormality detection device 13 of this embodiment. This embodiment is different from the third embodiment in that the equipment installation position information (equipment information) and the vehicle running position information F2 from the equipment database F1 are added to the equipment detection, and the equipment installation position information and the vehicle running position information F4 from the equipment database F3 are added to the part detection. Since the other operations and functions are the same, the same reference numerals are given to the drawings and the description thereof is omitted. Adding the vehicle running information and the equipment installation information to the equipment detection and the part detection means, for example, that it is possible to determine whether or not the equipment is installed from the vehicle position information of the captured image. As a result, in addition to the effects of the third embodiment, it is possible to improve the detection accuracy by suppressing overdetection in which it is determined that the equipment or parts are present when they are not present, and by ensuring detection at the running position where the equipment or parts are installed.

[0049] E. Fifth embodiment FIG. 15 shows a flowchart of the operation of the overhead line fitting abnormality detection device 13 of this embodiment. This embodiment differs from the first embodiment in that after equipment detection (FIG. 15, S003) or part detection (FIG. 15, S006), the detection result is displayed and the detection result is edited by the worker. Other than this, the operations and functions are the same, so the same reference numerals are given in the drawings and their explanations are omitted. In the operation of this embodiment, for example, the detection result is displayed on the display of the user interface (input / output unit) 14. The worker checks the detection result displayed on the display, and if there is an erroneous detection, manually corrects the rectangular position and / or label from the pointing device and / or keyboard of the user interface (input / output unit) 14 (FIG. 15, S501). As a result, in addition to the effects of the first embodiment, the image input to the abnormality judgment is corrected to be correct, and highly accurate abnormality judgment can be performed. The configuration of this embodiment in which an operator manually edits the detection results can also be applied to the second, third, and fourth embodiments, as shown in Figures 16, 17, and 18, respectively. In addition to the similar effects and advantages of each embodiment, the image input for anomaly judgment is corrected to make it correct, making it possible to perform highly accurate anomaly judgment.

[0050] F. Sixth embodiment FIG. 19 shows a flowchart of the operation of the overhead line fitting abnormality detection device 13 of this embodiment. The device configuration is the same as that shown in FIG. 2 in the first embodiment. This embodiment differs from the first embodiment in that image correction processing (FIG. 19, S601) and image inversion processing (FIG. 19, S602, S603) are performed after image cropping (FIG. 19, S007a) and image resizing (FIG. 19, S007b). Note that the step expressed as image cropping and resizing (FIG. 3, S007) in the flowchart of the first embodiment in FIG. 19 is expressed as two steps as image cropping (FIG. 19, S007a) and image resizing (FIG. 19, S007b), but the operation is the same as in the first embodiment. Other than these, the operation and function are the same as in the first embodiment, so the same reference numerals are used in the flowchart of FIG. 19 and the description thereof is omitted.

[0051] The following describes the new functions added in this embodiment. <Image correction processing> In the image correction process, an image is cut out based on the "rectangular coordinate information of equipment" indicated by dashed line A in Fig. 5 or the "rectangular coordinate information of parts" indicated by dashed line B in Fig. 7 (Fig. 19, S0007a), and the image data is resized to an image size that can be handled by the anomaly detection model (Fig. 19, S007b) and corrected (Fig. 19, S601). As the image correction process, for example, brightness correction process and edge enhancement process are performed. At this time, any method may be used for the brightness correction method and edge enhancement process, and examples include gamma correction and contrast correction for brightness correction, and bilateral filtering for edge enhancement. The corrected "corrected image" is sent to the next image inversion process together with the "rectangular coordinate information" and "label information".

[0052] <Image inversion processing> In the image inversion process, information on the region to be inverted horizontally is read from the database of region information on the region to be inverted horizontally (D20 in FIG. 19), and it is checked whether the position of the detected object in the corrected image data received from the previous image correction process (S601 in FIG. 19) belongs to the region to be inverted horizontally (S602 in FIG. 19). If the position of the detected object belongs to the region to be inverted horizontally (YES in S601 in FIG. 19), the image inversion process is performed (S603 in FIG. 19), and the inverted image data is sent to the abnormality degree calculation process (S008 in FIG. 19). If the detected position of the detected object does not belong to the region to be inverted horizontally (NO in S601 in FIG. 19), the corrected image data is sent directly to the abnormality degree calculation process (S008 in FIG. 19).

[0053] Here, the image inversion process will be described in more detail. FIG. 20 is a schematic diagram for explaining the situation in which the overhead wire fitting 3, which is the object to be inspected, is photographed in this embodiment. As shown in FIG. 20, a vehicle 1, which is a railroad vehicle, is running on a railroad track 2. A right camera 11a and a left camera 11b are installed on the roof of the vehicle 1. The right camera 11a and the left camera 11b are installed with their optical axes facing upward so as to photograph the overhead wire fitting 3 located above the vehicle 1. In FIG. 20, the overhead wire fitting 3 is exemplified by a suspension wire 3a, a hanger 3b, and a trolley wire 3c. FIG. 21 is a plan view of the vehicle 1 and the configuration including the right camera 11a and the left camera 11b shown in FIG. 20, seen from above. Rectangular images cut out as equipment from the images photographed by the right camera 11a and the left camera 11b are shown as images R and L. The white arrow S indicates the scanning direction of the right camera 11a and the left camera 11b facing upward. Here, the explanation focuses on the left camera, but a similar situation occurs with the right camera.

[0054] FIG. 22 shows image L0 before image cropping, captured by left camera 11b in the positional relationship of FIG. 21. Image L is cropped from image L0 by equipment detection processing. The area divided left and right by line C defines the above-mentioned left-right inversion target area. This area information is recorded in the left-right inversion target area information database (FIG. 19, D20). In FIG. 22, overhead line fitting 3 is captured on the right side of line C. As shown in FIG. 20 and FIG. 21, overhead line fitting 3 is located in the upper right corner on the paper surface of FIG. 21 when viewed from left camera 11b. Therefore, when the camera scans in the direction of arrow S, as shown in FIG. 22, the captured image captures contact wire 3c, hanger 3b, and suspension wire 3a in that order from the right side of the paper surface.

[0055] FIG. 23 is a plan view similar to FIG. 21, but shows that the overhead line fitting 3 is located in the upper left of the paper as viewed from the left camera 11b. FIG. 24 shows image L1 before extraction, captured by the left camera 11b in the positional relationship of FIG. 23. Image La is extracted from image L1 by equipment detection processing. Here, the overhead line fitting 3 is captured on the left side of the line C. As shown in FIG. 23, the overhead line fitting 3 is located in the upper left of the paper as viewed from the left camera 11b. Therefore, when the camera scans in the direction of the arrow S, the captured image captures the suspension wire 3a, hanger 3b, and trolley wire 3c in this order from the right of the paper, as shown in FIG. 24.

[0056] From the above considerations, the symmetry of the detection object changes depending on whether the detection object is photographed in the right region or the left region with respect to the line C. When learning an anomaly detection model and when detecting an anomaly using the model, it is better to align the configuration of the detection object because it is possible to improve detection accuracy by aligning the symmetry and executing learning and detection. For example, as shown in FIG. 22, when the images are aligned in the order of the trolley wire 3c, the hanger 3b, and the suspension wire 3a from the right of the paper, image processing is performed so that if the detection object is located in the right region of the line C, the left and right images are not inverted, and if the detection object is located in the left region of the line C, the left and right images are inverted. In this case, for the right camera 11a, the region in which the detection object is located is determined and the left and right images are inverted so that the image is similarly arranged from the right of the image to be the trolley wire 3c, the hanger 3b, and the suspension wire 3a.

[0057] FIG. 25 is a diagram showing the positional relationship of the overhead wire 20 with respect to the railway track 2. In FIG. 25, two railway tracks 2 and overhead wires 20 are shown, one on the left and one on the right. In the images captured by the right camera 11a and the left camera 11b, the overhead wire fittings 3 to be detected move left and right in the image area because, as shown in FIG. 25, the overhead wire 20 is installed so as to be pulled left and right with respect to the railway track 2 by the curve drawing device 23. The overhead wire 20 is arranged in this manner in order to prevent local wear of the pantograph, which is a power receiving device. That is, in order to make the contact frequency between the overhead wire 20 and the pantograph even on the contact surface of the pantograph, the overhead wire 20 is arranged in a so-called zigzag pattern. Therefore, even on the running line, the overhead wire 20 may straddle the center axis of the camera. In addition, when there is a railway track 2 (such as an up-down line) running parallel to the overhead wire 2, the overhead wire equipment of the parallel railway track 2 may be detected, and they may also be imaged on the reverse area side. In any case, the left and right images are inverted based on whether the object to be detected is in an image inversion area, and the image inversion processing is performed so that the images provided for calculating the degree of abnormality have a common configuration (the arrangement of the suspension wire, hanger, and trolley wire is constant).

[0058] The process following the image inversion process in this embodiment is the same as that in the first embodiment. However, the calculation of the degree of anomaly is applied to the cut-out image that has been subjected to image correction and inversion processing, and the degree of anomaly is calculated. At this time, the pre-learning model performs learning using data that has been subjected to image correction processing and image inversion processing.

[0059] As described above, in this embodiment, the image correction process and the image inversion process are performed on the cut and resized image data, so that in addition to the actions and effects of the first embodiment, the accuracy of anomaly detection can be improved. Specifically, brightness correction process and edge enhancement process are performed on all data as image correction processes on the cut and resized images after equipment detection. This makes it possible to realize highly accurate anomaly detection even in cases where the target equipment is not clearly photographed, such as in environments where low brightness is likely to be captured, such as at night or inside a tunnel. In addition, in this embodiment, the image inversion process can align the order of detection targets in the image data for calculating the degree of anomaly, thereby improving the accuracy of anomaly detection.

[0060] G. Seventh embodiment FIG. 26 shows a flowchart of the operation of the overhead line fitting abnormality detection device 13 of this embodiment. The device configuration is the same as that of FIG. 2 shown in the first embodiment. This embodiment differs from the sixth embodiment in that an image correction process (FIG. 26, S701) is performed on all captured images. Other than this, the operations and functions are the same as those of the sixth embodiment, so that the same reference numerals are used in the flowchart of FIG. 26 and their explanations are omitted. In this embodiment, in addition to the actions and effects of the sixth embodiment, brightness correction process and edge enhancement process are added as image correction process before equipment detection, so that highly accurate equipment detection and abnormality detection can be realized even in cases where the target equipment is not clearly captured, such as in an environment where the equipment is likely to be captured at low brightness, such as at night or inside a tunnel.

[0061] H. Eighth embodiment FIG. 27 shows a flowchart of the operation of the overhead wire fitting abnormality detection device 13 of this embodiment. The device configuration is the same as that of FIG. 2 shown in the first embodiment. This embodiment differs from the sixth embodiment in that, instead of the image correction process of the sixth embodiment (FIG. 19, S601), correction is performed on the image after image cropping and resizing based on the shooting time and the vehicle's running position information (FIG. 27, S801, S802). Since the other operations and functions are the same as those of the sixth embodiment, the same reference numerals are given in the flowchart of FIG. 27 and the description thereof is omitted. Specifically, in the sixth embodiment, the image correction process is performed on all cropped and resized image data, but based on the target shooting time and running information (position information of the point where the vehicle is currently running is acquired from the vehicle), a running section that may become unclear, such as at night or inside a tunnel, is determined (determined against the currently running position information based on the previously acquired mileage information of the tunnel, etc.), and the image data to which the image correction is applied is limited and the process is applied.

[0062] The functions provided in this embodiment will be described below. <Blurred area determination> When the cropped and resized image is supplied, the data processing unit 15 refers to the shooting time and driving position information, and if it is determined based on the information that the driving section is one that may be blurred, such as at night or inside a tunnel, a flag is set to execute image correction processing (FIG. 27, S801). If it is determined that the section is not blurred (FIG. 27, S801, NO), the process proceeds to the next left-right image inversion processing (FIG. 27, S602). If it is determined that the section is blurred (FIG. 27, S801, YES), the image correction processing is performed (FIG. 27, S802). The image correction processing here is the same as S601 in FIG. 19 of the sixth embodiment. After the image correction is completed, the process proceeds to the next determination of left-right image inversion processing (FIG. 27, S602).

[0063] In this embodiment, in addition to the actions and effects of the sixth embodiment, brightness correction and edge enhancement are performed as image correction processes on the cropped and resized image after facility detection, based on the shooting time and driving information. Therefore, when the target facility is not clearly captured, such as in an environment where it is easy to capture low-luminance images, such as at night or inside a tunnel, image correction is performed by accurately determining the case, thereby realizing highly accurate anomaly detection.

[0064] I. Ninth embodiment The flow chart of the operation of the overhead line fitting abnormality detection device 13 of this embodiment is the same as that shown in Fig. 27 in the eighth embodiment. The device configuration is also the same as that shown in Fig. 2 in the first embodiment. This embodiment differs from the eighth embodiment in the condition judgment for judging the unclear section. Other than this, the operation and functions are the same as those of the eighth embodiment, so a description thereof will be omitted.

[0065] In this embodiment, the conditions for determining the blurred section are based on the brightness information of the image data instead of the shooting time and the vehicle running position information of the eighth embodiment. For example, an automatic determination process is used to determine the running section that may become blurred at night or inside a tunnel. In the automatic determination process, the determination may be made by an algorithm using brightness information such as the average brightness of the captured image (determining each line or each image obtained by combining multiple lines by "average brightness<threshold"). Alternatively, the determination may be made using a discrimination AI (AlexNet, VGG, GoogleNet, ResNet, etc.). In addition, when an algorithm based on brightness information is used, the average value and variance of brightness may be used. When statistical information of brightness is used, preprocessing such as noise removal may be performed as necessary before calculating the statistical information.

[0066] In this embodiment, in addition to the actions and effects of the sixth embodiment, brightness correction and edge enhancement are performed as image correction processes on the cropped and resized images after equipment detection, based on the image luminance information and classification AI. Therefore, when the target equipment is not clearly photographed, such as in environments where it is easy to photograph with low luminance, such as at night or inside a tunnel, image correction is performed by accurately determining the case, and highly accurate anomaly detection can be realized.

[0067] In addition, it is possible to select and discard the configurations given in the above embodiment, or to change them to other configurations as appropriate, without departing from the spirit of the present invention. [Explanation of symbols]

[0068] 1 vehicle 2. Railway tracks 3. Overhead Wiring Fittings 11 Camera 12 Storage device 13. Cable fitting abnormality detection device 14 User interface (input / output section) 15 Data Processing Section 17 Learning processing section 19 Facilities Learning Department 21 Parts Learning Section

Claims

1. A learning processing unit that performs learning using image data generated by photographing the overhead line fittings; a data processing unit that performs an abnormality determination using the image data, The learning processing unit includes an equipment learning unit that learns to detect equipment, and a part learning unit that learns to detect parts of the equipment, The data processing unit detects the equipment using an equipment detection model learned by the equipment learning unit, detects the parts from the detected equipment using a part detection model learned by the part learning unit, and determines an abnormality in the equipment and / or the parts using an anomaly detection model learned in advance; The equipment detection model and the part detection model are selected from models that do not use semantic segmentation, including a Single Shot Multibox Detector (SSD).

2. 2. The overhead line fitting abnormality detection device according to claim 1, wherein the data processing unit detects the equipment based on the equipment detection model, vehicle running position information, and equipment information.

3. 3. The overhead line fitting abnormality detection device according to claim 1, wherein the data processing unit determines an abnormality in equipment and / or components based on a restoration error of the abnormality detection model.

4. 3. The overhead line fitting abnormality detection device according to claim 1 or 2, further comprising an input / output unit, wherein detection results of the equipment and / or the parts detected by the data processing unit are displayed on the input / output unit, and an operator can check and edit the detection results via the input / output unit.

5. Learning using image data generated by photographing overhead line fittings; performing an abnormality determination using the image data; The learning includes learning to detect equipment and learning to detect parts of the equipment; The performing of the anomaly determination includes detecting the equipment using an equipment detection model trained by learning to detect the equipment, detecting the parts from the detected equipment using a part detection model trained by learning to detect parts of the equipment, and determining an anomaly of the equipment and / or the parts using a previously trained anomaly detection model; The overhead line fitting anomaly detection method, wherein the equipment detection model and the part detection model are selected from models that do not use semantic segmentation, including a Single Shot Multibox Detector (SSD).

6. 6. The overhead line fitting abnormality detection method according to claim 5, wherein the determination of the abnormality includes detecting the equipment based on the equipment detection model, vehicle running position information, and equipment information.

7. 7. The overhead line fitting anomaly detection method according to claim 5, wherein the performing of the anomaly determination includes determining an anomaly in equipment and / or a component based on a restoration error of the anomaly detection model.

8. 7. The overhead line fitting abnormality detection method according to claim 5 or 6, further comprising: displaying the detection results of the equipment and / or the component detected by performing the abnormality judgment on an input / output unit; and having an operator check and edit the detection results from the input / output unit.

9. A learning processing unit that performs learning using image data generated by photographing the overhead line fittings; a data processing unit that performs an abnormality determination using the image data, The learning processing unit includes an equipment learning unit that learns to detect equipment, and a part learning unit that learns to detect parts of the equipment, In the data processing unit, Detecting the facility using a facility detection model learned by the facility learning unit; Detecting the parts from the detected equipment using the part detection model learned by the part learning unit; Correcting the image data in which the equipment or the part is detected; Anomalies in equipment and / or parts are determined using a pre-trained anomaly detection model, The equipment detection model and the part detection model are selected from models that do not use semantic segmentation, including a Single Shot Multibox Detector (SSD).

10. A learning processing unit that performs learning using image data generated by photographing the overhead line fittings; a data processing unit that performs an abnormality determination using the image data, The learning processing unit includes an equipment learning unit that learns to detect equipment, and a part learning unit that learns to detect parts of the equipment, In the data processing unit, Correcting the image data; Detecting the facility using a facility detection model learned by the facility learning unit; Detecting the parts from the detected equipment using the part detection model learned by the part learning unit; Anomalies in equipment and / or parts are determined using a pre-trained anomaly detection model, The equipment detection model and the part detection model are selected from models that do not use semantic segmentation, including a Single Shot Multibox Detector (SSD).

11. A learning processing unit that performs learning using image data generated by photographing the overhead line fittings; a data processing unit that performs an abnormality determination using the image data, The learning processing unit includes an equipment learning unit that learns to detect equipment, and a part learning unit that learns to detect parts of the equipment, In the data processing unit, Detecting the facility using a facility detection model learned by the facility learning unit; Detecting the parts from the detected equipment using the part detection model learned by the part learning unit; correcting the image data in which the facility or the part is detected based on the photographing time and vehicle running position information; Anomalies in equipment and / or parts are determined using a pre-trained anomaly detection model, The equipment detection model and the part detection model are selected from models that do not use semantic segmentation, including a Single Shot Multibox Detector (SSD).

12. A learning processing unit that performs learning using image data generated by photographing the overhead line fittings; a data processing unit that performs an abnormality determination using the image data, The learning processing unit includes an equipment learning unit that learns to detect equipment, and a part learning unit that learns to detect parts of the equipment, In the data processing unit, Detecting the facility using a facility detection model learned by the facility learning unit; Detecting the parts from the detected equipment using the part detection model learned by the part learning unit; Correcting the image data based on brightness information or identification AI of the image data in which the equipment or the part is detected; Anomalies in equipment and / or parts are determined using a pre-trained anomaly detection model, The equipment detection model and the part detection model are selected from models that do not use semantic segmentation, including a Single Shot Multibox Detector (SSD).

13. In the data processing unit, 13. The overhead line fitting abnormality detection device according to claim 9, further comprising: an inversion process for the image data based on a position of a detection object in the image data before determining an abnormality in equipment and / or components using the pre-trained abnormality detection model.

14. Learning using image data generated by photographing overhead line fittings; performing an abnormality determination using the image data; The learning includes learning to detect equipment and learning to detect parts of the equipment; The abnormality determination is Detecting the facility using a facility detection model trained by learning to detect the facility; Detecting the parts using a part detection model trained by learning to detect parts of the equipment from the detected equipment; correcting the image data in which the equipment or the part is detected; Determining an anomaly in the equipment and / or the part using a pre-trained anomaly detection model; The overhead line fitting anomaly detection method, wherein the equipment detection model and the part detection model are selected from models that do not use semantic segmentation, including a Single Shot Multibox Detector (SSD).

15. Learning using image data generated by photographing overhead line fittings; performing an abnormality determination using the image data; The learning includes learning to detect equipment and learning to detect parts of the equipment; The abnormality determination is correcting the image data; Detecting the facility using a facility detection model trained by learning to detect the facility; Detecting the parts using a part detection model trained by learning to detect parts of the equipment from the detected equipment; Determining an anomaly in the equipment and / or the part using a pre-trained anomaly detection model; The overhead line fitting anomaly detection method, wherein the equipment detection model and the part detection model are selected from models that do not use semantic segmentation, including a Single Shot Multibox Detector (SSD).

16. Learning using image data generated by photographing overhead line fittings; performing an abnormality determination using the image data; The learning includes learning to detect equipment and learning to detect parts of the equipment; The abnormality determination is Detecting the facility using a facility detection model trained by learning to detect the facility; Detecting the parts using a part detection model trained by learning to detect parts of the equipment from the detected equipment; correcting the image data in which the facility or the part is detected based on the photographing time and vehicle running position information; Determining an anomaly in the equipment and / or the part using a pre-trained anomaly detection model; The overhead line fitting anomaly detection method, wherein the equipment detection model and the part detection model are selected from models that do not use semantic segmentation, including a Single Shot Multibox Detector (SSD).

17. Learning using image data generated by photographing overhead line fittings; performing an abnormality determination using the image data; The learning includes learning to detect equipment and learning to detect parts of the equipment; The abnormality determination is Detecting the facility using a facility detection model trained by learning to detect the facility; Detecting the parts using a part detection model trained by learning to detect parts of the equipment from the detected equipment; Correcting the image data based on brightness information or identification AI of the image data in which the equipment or the part is detected; Determining an anomaly in the equipment and / or the part using a pre-trained anomaly detection model; The overhead line fitting anomaly detection method, wherein the equipment detection model and the part detection model are selected from models that do not use semantic segmentation, including a Single Shot Multibox Detector (SSD).

18. The abnormality determination is The overhead line fitting abnormality detection method according to any one of claims 14 to 17, further comprising: performing an inversion process on the image data based on a position of a detection target object in the image data before determining an abnormality in equipment and / or components using the pre-trained anomaly detection model.

Images